Oct 30 18:36:10.903359 ip-10-0-134-34 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:11.352264 ip-10-0-134-34 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:11.352264 ip-10-0-134-34 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:11.352264 ip-10-0-134-34 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:11.352264 ip-10-0-134-34 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:11.359224 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359131    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.359224 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359218    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.359224 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359223    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.359224 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359228    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.359224 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359235    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359239    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359243    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359246    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359249    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359252    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359255    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359258    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359261    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359263    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359268    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359271    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359274    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359277    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359280    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359283    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359286    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359289    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359291    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.359430 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359294    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359297    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359299    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359302    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359305    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359307    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359310    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359313    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359321    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359324    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359326    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359329    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359331    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359334    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359336    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359339    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359342    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359346    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359349    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.359878 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359352    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359355    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359357    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359360    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359362    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359365    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359368    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359370    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359373    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359376    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359378    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359381    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359384    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359386    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359389    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359392    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359395    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359398    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359401    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.360346 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359403    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359406    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359409    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359412    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359415    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359417    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359420    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359423    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359425    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359429    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359432    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359435    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359438    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359440    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359443    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359446    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359449    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359452    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359454    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359457    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.360796 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359459    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359462    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359466    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359469    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359472    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359474    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359477    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359480    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359482    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359486    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359489    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359491    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359494    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359497    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359499    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359502    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359504    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359507    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359509    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359512    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.361296 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359514    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359518    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359521    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359524    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.359527    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360589    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360599    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360603    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360607    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360610    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360614    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360617    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360620    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360622    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360625    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360628    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360631    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360634    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360637    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.361793 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360639    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360642    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360645    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360647    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360650    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360653    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360655    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360658    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360660    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360663    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360665    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360668    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360671    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360674    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360677    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360680    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360683    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360686    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360689    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360692    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.362278 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360694    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360697    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360700    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360702    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360705    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360708    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360710    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360713    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360715    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360719    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360721    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360724    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360726    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360729    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360732    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360734    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360737    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360739    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360742    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360744    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.362760 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360747    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360749    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360752    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360754    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360757    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360760    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360763    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360765    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360768    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360770    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360773    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360776    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360778    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360782    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360785    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360787    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360790    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360793    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360795    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.363252 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360801    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360803    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360807    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360809    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360812    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360815    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360817    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360820    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360822    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360825    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360827    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360830    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360832    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360835    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360837    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360840    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360843    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360847    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360851    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.363734 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360854    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360856    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360859    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360862    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360865    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360868    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360873    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360877    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360880    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360883    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360885    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360888    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.360890    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361416    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361422    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361426    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361428    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361431    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361434    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.364202 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361437    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361440    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361442    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361445    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361447    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361450    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361453    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361456    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361458    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361461    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361464    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361467    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361470    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361472    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361475    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361478    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361480    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361483    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361485    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.364635 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361488    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361491    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361493    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361496    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361499    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361501    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361504    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361507    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361509    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361512    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361515    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361519    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361522    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361525    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361528    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361531    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361534    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361537    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361540    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361543    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.365096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361545    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361553    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361559    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361561    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361564    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361567    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361569    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361572    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361575    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361577    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361580    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361583    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361585    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361588    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361590    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361593    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361596    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361598    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361601    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361604    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.365567 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361606    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361608    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361611    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361614    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361617    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361619    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361621    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361626    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361629    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361633    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361635    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361638    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361641    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361643    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361647    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361650    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361652    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361655    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361657    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.366057 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361660    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361662    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361665    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361667    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361670    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361672    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361675    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361677    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361680    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361682    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361685    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361687    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361690    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361692    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361695    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361697    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361700    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361708    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361711    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361720    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.366553 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361724    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.361729    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363663    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363673    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363680    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363685    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363690    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363694    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363699    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363703    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363706    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363710    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363713    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363716    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363719    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363722    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363725    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363729    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363731    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363735    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363739    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363742    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363745    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363749    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:11.367031 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363752    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363756    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363759    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363762    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363766    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363769    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363776    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363780    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363783    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363787    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363791    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363794    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363797    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363800    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363803    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363808    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363813    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363816    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363820    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363823    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363827    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363831    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363834    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363838    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363841    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:11.367597 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363844    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363847    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363850    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363853    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363856    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363859    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363862    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363867    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363870    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363873    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363876    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363879    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363882    2579 flags.go:64] FLAG: --help="false"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363885    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363889    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363892    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363896    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363899    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363902    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363906    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363909    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363911    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363916    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:11.368312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363919    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363922    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363925    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363928    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363931    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363934    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363937    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363940    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363942    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363945    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363949    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363955    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363958    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363961    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363964    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363967    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363971    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363974    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363977    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363981    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363984    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363988    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363992    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.363995    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364001    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:11.368858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364016    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364019    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364022    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364025    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364033    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364036    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364041    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364044    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364047    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364051    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364054    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364057    2579 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364060    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364063    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02f0857ff1c50eacf"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364066    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364069    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364072    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364075    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364079    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364082    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364085    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364088    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364091    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364094    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364097    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364100    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:11.369481 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364104    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364107    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364110    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364113    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364116    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364119    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364126    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364130    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364133    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364136    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364139    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364142    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364146    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364149    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364152    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364155    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364161    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364163    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364166    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364170    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364173    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364176    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364179    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364182    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364185    2579 flags.go:64] FLAG: --v="2"
Oct 30 18:36:11.370177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364189    2579 flags.go:64] FLAG: --version="false"
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364194    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364198    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364201    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364362    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364366    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364369    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364374    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364377    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364379    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364382    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364385    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364387    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364390    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364398    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364401    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364404    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364407    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364409    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364414    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.370789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364417    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364420    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364422    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364425    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364428    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364430    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364433    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364435    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364438    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364440    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364443    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364447    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364451    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364454    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364456    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364459    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364462    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364464    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364467    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.371404 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364471    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364473    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364476    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364479    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364482    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364484    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364487    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364489    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364497    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364500    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364503    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364505    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364509    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364512    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364515    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364517    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364520    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364523    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364526    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.371850 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364528    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364531    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364533    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364536    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364539    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364542    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364545    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364547    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364549    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364552    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364555    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364557    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364560    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364564    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364566    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364569    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364572    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364574    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364577    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364579    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.372319 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364582    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364585    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364593    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364596    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364599    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364603    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364606    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364609    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364611    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364614    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364617    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364619    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364622    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364624    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364627    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364630    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364632    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364635    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364637    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.372789 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364640    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364643    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364646    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364649    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364651    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364654    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364657    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364660    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364663    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364667    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364671    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364674    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364678    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.364705    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364742    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364747    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.373249 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364751    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364754    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364757    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364760    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364763    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364766    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364768    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364771    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364773    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364776    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364779    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364782    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364784    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364787    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364789    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364792    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364794    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364797    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364800    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.373625 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364802    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364805    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364807    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364811    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364814    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364816    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364819    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364822    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364824    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364827    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364830    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364832    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364835    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364838    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364841    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364844    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364847    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364849    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364852    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364854    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.374096 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364857    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364859    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364862    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364865    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364867    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364870    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364873    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364875    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364878    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364880    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364883    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364885    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364888    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364890    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364893    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364897    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364899    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364902    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364905    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364907    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.374552 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364910    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364912    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364915    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364917    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364922    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364925    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364928    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364930    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364933    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364935    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364939    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364942    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364946    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364948    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364951    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364954    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364957    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364959    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364962    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.375033 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364964    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364967    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364969    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364972    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364974    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364977    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364979    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364983    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364985    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364988    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364991    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364993    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364996    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.364998    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365001    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365003    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365017    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365019    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365022    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.375541 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365024    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365027    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365030    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365038    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365042    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.365045    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.372616    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.372634    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.373367    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.373519    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373619    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373628    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373635    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373640    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373645    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373650    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373654    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373658    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373662    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.375993 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373667    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373670    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373673    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373675    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373678    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373681    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373684    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373686    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373689    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373691    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373694    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373697    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373700    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373702    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373705    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373707    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373710    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373713    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373724    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373727    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.376463 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373741    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373744    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373747    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373750    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373753    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373756    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373758    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373761    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373764    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373766    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373769    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373771    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373774    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373776    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373779    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373782    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373785    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373787    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373790    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.376945 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373792    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373795    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373800    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373803    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373806    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373808    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373811    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373813    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373816    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373818    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373821    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373824    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373827    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373835    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373838    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373841    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373844    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373846    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373849    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.377481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373851    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373854    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373856    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373859    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373864    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373868    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373872    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373875    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373878    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373881    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373884    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373887    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373890    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373893    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373897    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373899    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373902    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373904    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373907    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.377967 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373909    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373912    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373915    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373917    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373920    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373923    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373926    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373935    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373939    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373943    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373947    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373950    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373953    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373956    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373958    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373961    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373980    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.378454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.373984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.374037    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374240    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374246    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374249    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374251    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374254    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374257    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374260    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374263    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374266    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374269    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374271    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374274    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374276    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374279    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.378895 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374281    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374284    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374286    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374289    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374294    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374297    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374305    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374308    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374311    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374313    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374316    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374319    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374322    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374324    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374327    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374329    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374333    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374335    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374338    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.379295 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374340    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374343    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374346    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374348    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374351    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374354    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374356    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374359    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374361    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374364    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374367    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374369    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374372    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374375    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374377    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374380    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374382    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374385    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374387    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.379784 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374390    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374398    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374401    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374403    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374406    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374408    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374411    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374413    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374416    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374419    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374422    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374424    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374427    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374429    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374432    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374434    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374437    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374439    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374442    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374444    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.380253 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374447    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374455    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374458    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374461    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374465    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374468    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374472    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374474    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374477    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374480    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374483    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374487    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374491    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374494    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374502    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374505    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374507    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374510    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374513    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.380727 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374515    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374518    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374520    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374523    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374526    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374528    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374530    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374533    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374535    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374538    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374540    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374543    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374545    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374548    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.374571    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:11.381190 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374609    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374613    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374616    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374619    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374621    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374624    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374627    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374630    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374633    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374636    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374639    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374641    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374644    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374652    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374655    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374657    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374660    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374662    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374665    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:11.381551 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374668    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374670    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374673    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374675    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374678    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374680    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374683    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374685    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374688    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374690    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374693    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374695    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374698    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374700    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374703    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374706    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374708    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374710    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374713    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374716    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:11.382094 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374720    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374724    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374727    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374730    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374733    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374736    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374738    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374746    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374750    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374754    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374756    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374760    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374762    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374765    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374767    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374770    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374772    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374775    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374777    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:11.382781 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374781    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374783    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374786    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374788    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374791    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374793    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374796    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374798    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374800    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374803    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374805    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374808    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374811    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374813    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374816    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374819    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374821    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374824    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374827    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374829    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:11.383540 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374832    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374840    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374843    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374845    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374848    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374851    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374853    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374856    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374858    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374860    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374863    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374865    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374868    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374870    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374873    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374875    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374878    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374880    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374883    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374885    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:11.384266 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374887    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374890    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374893    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374895    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374898    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374901    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374903    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:11.374907    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.374972    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.377487    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.379578    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.379688    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.379735    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:11.384744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.382044    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:11.385359 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.385340    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:11.395225 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.395198    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:11.399459 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.399439    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:11.400308 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.400280    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:11.407235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.407209    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c083c8e-20a4-43e0-84f1-3614c4fe40f1:/dev/nvme0n1p3 8543e099-c652-4ddc-b640-72590b55d365:/dev/nvme0n1p4]
Oct 30 18:36:11.407235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.407231    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:36:11.412990 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.412866    2579 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:11.411302138 +0000 UTC m=+0.375344924 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095922 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d2000e31f1afacf01c497603e72c SystemUUID:ec27d200-0e31-f1af-acf0-1c497603e72c BootID:c8994aaa-a7aa-499f-9f00-230a502bcb7b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:21:e4:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:21:e4:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:78:d4:4d:ab:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:11.412990 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.412986    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:11.413139 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.413126    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:11.416392 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.416356    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:11.416581 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.416396    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-34.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:11.416673 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.416584    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:11.416673 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.416592    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:11.416673 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.416608    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:11.417490 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.417478    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:11.418783 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.418772    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:11.423303 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423285    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:11.423436 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423427    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:11.423477 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423461    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:11.423505 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423478    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:11.423505 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423491    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:11.423505 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423500    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:11.423598 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.423509    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:11.424242 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.424228    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:11.426499 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.426464    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:11.428179 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.428155    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:11.433505 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.433488    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:11.434451 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.434439    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:11.434503 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.434469    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:11.435328 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435317    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:11.435383 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435332    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:11.435383 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435340    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:11.435383 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435345    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:11.435383 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435351    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:11.435383 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.435357    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:11.436071 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436062    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:11.436071 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436072    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:11.436125 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436079    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:11.436125 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436085    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:11.436125 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436095    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:11.436125 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436103    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:11.436948 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436938    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:11.436948 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.436948    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:11.439923 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.439899    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-34.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:36:11.441105 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.441090    2579 server.go:1263] "Started kubelet"
Oct 30 18:36:11.441229 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.441197    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:11.441268 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.441202    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:11.441301 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.441274    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:11.442045 ip-10-0-134-34 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:11.442944 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.442919    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:11.443963 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.443945    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:11.444837 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.444808    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-34.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:36:11.445307 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.445037    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:36:11.447608 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.447581    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:36:11.452334 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.452297    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:11.452943 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.452925    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:11.453096 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.453062    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.453803 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.453781    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:11.454067 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454045    2579 factory.go:55] Registering systemd factory
Oct 30 18:36:11.454149 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454072    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:11.454204 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454175    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:11.454204 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454192    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:11.454393 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454374    2579 factory.go:153] Registering CRI-O factory
Oct 30 18:36:11.454455 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454399    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:11.454515 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454456    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:11.454515 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454476    2579 factory.go:103] Registering Raw factory
Oct 30 18:36:11.454515 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454492    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:11.454793 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454778    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:11.454793 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.454791    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:11.455123 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.453960    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-34.us-east-2.compute.internal.18e3627e5a205a2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-34.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-34.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:11.441060397 +0000 UTC m=+0.405103181,LastTimestamp:2026-10-30 18:36:11.441060397 +0000 UTC m=+0.405103181,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-34.us-east-2.compute.internal,}"
Oct 30 18:36:11.455557 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.455543    2579 manager.go:319] Starting recovery of all containers
Oct 30 18:36:11.457952 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.457919    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-34.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:36:11.465087 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.464303    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:36:11.467088 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.467055    2579 manager.go:324] Recovery completed
Oct 30 18:36:11.473559 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.473544    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.475538 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.475512    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.475607 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.475551    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.475607 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.475563    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.476203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.476181    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:11.476203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.476196    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.476341 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.476198    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:11.476341 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.476260    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:11.477886 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.477867    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.477982 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.477898    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.477982 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.477908    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.478426 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.478350    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-34.us-east-2.compute.internal.18e3627e5c2e6b9b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-34.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-34.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-34.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:11.475536795 +0000 UTC m=+0.439579586,LastTimestamp:2026-10-30 18:36:11.475536795 +0000 UTC m=+0.439579586,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-34.us-east-2.compute.internal,}"
Oct 30 18:36:11.479801 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.479787    2579 policy_none.go:50] "Start"
Oct 30 18:36:11.479856 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.479805    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:11.479856 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.479816    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:11.481881 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.481865    2579 policy_none.go:44] "Start"
Oct 30 18:36:11.485321 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.485299    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88wxv"
Oct 30 18:36:11.493763 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.493647    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-34.us-east-2.compute.internal.18e3627e5c2eb6d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-34.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-34.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-34.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:11.475556054 +0000 UTC m=+0.439598838,LastTimestamp:2026-10-30 18:36:11.475556054 +0000 UTC m=+0.439598838,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-34.us-east-2.compute.internal,}"
Oct 30 18:36:11.497808 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.497655    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88wxv"
Oct 30 18:36:11.510911 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.510858    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:11.515618 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.515599    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.515631    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.515643    2579 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.516095    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.516109    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.516213    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.516347    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.516357    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.517182    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.517234    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.533845    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.535139    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.535168    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.535198    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:11.537327 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.535244    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:36:11.541965 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.541941    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:11.616921 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.616793    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.617927 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.617909    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.618050 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.617939    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.618050 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.617953    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.618050 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.617986    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.628449 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.628424    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.628532 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.628455    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-34.us-east-2.compute.internal\": node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.635588 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.635565    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"]
Oct 30 18:36:11.635672 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.635628    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.636534 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.636520    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.636607 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.636550    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.636607 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.636565    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.638777 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.638765    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.638909 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.638895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.638953 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.638924    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.639503 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639489    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.639571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639504    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.639571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639514    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.639571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639524    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.639571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639530    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.639571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.639545    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.641664 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.641650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.641729 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.641675    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:11.642466 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.642450    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:11.642547 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.642480    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:11.642547 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.642495    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:11.648229 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.648209    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.655837 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.655814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.655974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.655847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.655974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.655909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.676847 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.676826    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.678839 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.678816    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.748890 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.748863    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.756574 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.756551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.756668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.756579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.756668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.756616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.756668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.756645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.756668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.756655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.757118 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.757096    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.849744 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.849685    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.950508 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:11.950438    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:11.981992 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.981967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:11.982056 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:11.981992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:12.051572 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.051548    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.152105 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.152069    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.252583 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.252550    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.353042 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.352992    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.355312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.355290    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:12.379561 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.379523    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:36:12.379760 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.379709    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:12.379760 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.379717    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:12.453163 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.453129    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.503610 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.503507    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:11 +0000 UTC" deadline="2028-07-17 16:28:44.729965818 +0000 UTC"
Oct 30 18:36:12.503610 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.503550    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15021h52m32.226419664s"
Oct 30 18:36:12.554382 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.554197    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.572623 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.572598    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:12.573797 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.573776    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:12.573896 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.573828    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:12.573896 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.573841    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:12.655273 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:12.655248    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:12.656481 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:12.656455    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76487939913b7836b4b3b35bf9597886.slice/crio-42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd WatchSource:0}: Error finding container 42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd: Status 404 returned error can't find the container with id 42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd
Oct 30 18:36:12.657020 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:12.656998    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d63ad00e5a9245d4df7ea9f3aeb927.slice/crio-a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6 WatchSource:0}: Error finding container a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6: Status 404 returned error can't find the container with id a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6
Oct 30 18:36:12.663974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.663957    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:12.686572 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.686554    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:12.754399 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.754319    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:12.765272 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.765245    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:12.766389 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.766376    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:12.779097 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.779074    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:12.796316 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:12.796292    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:13.349792 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.349760    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:13.424479 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.424447    2579 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:13.432605 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.432577    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:13.432967 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.432945    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-x5hb6","openshift-multus/multus-additional-cni-plugins-bxfx7","openshift-multus/network-metrics-daemon-sfglz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc","openshift-cluster-node-tuning-operator/tuned-8z7tf","openshift-multus/multus-94k6t","openshift-network-operator/iptables-alerter-wqk77","openshift-ovn-kubernetes/ovnkube-node-njg9l","kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal","openshift-dns/node-resolver-flzfd","openshift-image-registry/node-ca-96jgv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-lz7bk","kube-system/global-pull-secret-syncer-qw889"]
Oct 30 18:36:13.435478 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.435459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.437920 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.437904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:13.437996 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.437976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:13.439921 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.439899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.440035 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.439971    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:13.440984 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.440964    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.441178 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.441130    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-psdnm\""
Oct 30 18:36:13.441178 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.441140    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.442270 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.442256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.444832 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.444504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.446824 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.446801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.449440 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.449420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.451627 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.451584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.453316 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.453296    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:13.456097 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.456079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.459161 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.459141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.465178 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.462076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.465239 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.465210    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:13.465902 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.465876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.465995 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.465922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.465995 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.465953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g28g\" (UniqueName: \"kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.465995 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.465984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.466235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.466235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.466235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.466235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.466235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skln\" (UniqueName: \"kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.466714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.466993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.467795 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8nq\" (UniqueName: \"kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbqz\" (UniqueName: \"kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.467990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.468549 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljdmf\" (UniqueName: \"kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5p7s\" (UniqueName: \"kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.469345 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559c2\" (UniqueName: \"kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cntt\" (UniqueName: \"kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.470128 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.468810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.488738 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.488634    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4h7b"
Oct 30 18:36:13.532551 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.532529    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4h7b"
Oct 30 18:36:13.540144 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.540100    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"76487939913b7836b4b3b35bf9597886","Type":"ContainerStarted","Data":"42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd"}
Oct 30 18:36:13.541408 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.541389    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.541504 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.541425    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.541957 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.541931    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:13.542110 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542095    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:13.542301 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.542381 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542281    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerStarted","Data":"a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6"}
Oct 30 18:36:13.542381 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542374    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:13.542491 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542388    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:13.542491 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542418    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.542491 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542450    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542500    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542500    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542549    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542562    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542609    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.542629 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542614    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542649    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542653    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542656    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542699    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542701    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rwngh\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542745    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542748    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542849    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.542893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542864    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.543236 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.542975    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4k7wf\""
Oct 30 18:36:13.543236 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543061    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543525    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jfgwk\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543544    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543554    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mkp5v\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543595    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdfmc\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543604    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9gnp\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543611    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mrncz\""
Oct 30 18:36:13.544195 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.543677    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-whw2z\""
Oct 30 18:36:13.554751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.554731    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:13.569753 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.569859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.569859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.569974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.569974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.569974 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570145 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.569985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570145 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570145 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.570145 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbqz\" (UniqueName: \"kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.570145 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdmf\" (UniqueName: \"kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.570384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5p7s\" (UniqueName: \"kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.570762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.570938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559c2\" (UniqueName: \"kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cntt\" (UniqueName: \"kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.571245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.571853 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.571853 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571269    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571853 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571853 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571376    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.571853 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.572122 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.572122 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.571939    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:13.572122 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572096    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.572286 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572152    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.572286 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572205    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.572460 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572439    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.572577 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572557    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.572668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572600    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.572718 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.572808 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572794    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.572845 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.572827    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.573114 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.573095    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.573202 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.573136    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.573408 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.573387    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.573634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.573608    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.574247 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.574219    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.574333 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.574280    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.574433 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.573190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.574433 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.574428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.574723 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.574706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.575750 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.575750 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575601    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.575750 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575701    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575766    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575797    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.575877    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575911    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.575968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.575990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g28g\" (UniqueName: \"kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.576306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576730 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.576730 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576837 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.576837 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.576837 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.576968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576843    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.576968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576887    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.576968 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.576918    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577094 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.576979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.076960404 +0000 UTC m=+3.041003199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:13.577199 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577172    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.577260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577206    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.577329 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.577388 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.577388 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.577483 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577483 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.577483 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577446    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577483 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577525    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skln\" (UniqueName: \"kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.577668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqnh\" (UniqueName: \"kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.577985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.577976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8nq\" (UniqueName: \"kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578129    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578237    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578336    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578401    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578439    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.578467 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578468    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578473    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578530    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578775    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578819    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.579156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.578888    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.579450 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.579269    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.581813 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.581793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.589865 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.589002    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:13.589865 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.589039    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:13.589865 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.589051    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:13.589865 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.589102    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.089085647 +0000 UTC m=+3.053128435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:13.589865 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.589297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdmf\" (UniqueName: \"kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.590583 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.590420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5p7s\" (UniqueName: \"kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.591720 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.591671    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8nq\" (UniqueName: \"kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:13.591809 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.591713    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g28g\" (UniqueName: \"kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.592130 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.592112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skln\" (UniqueName: \"kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.592130 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.592122    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cntt\" (UniqueName: \"kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.593849 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.593824    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559c2\" (UniqueName: \"kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.593933 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.593908    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbqz\" (UniqueName: \"kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.678962 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.678885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.678962 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.678933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679001    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.679177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.679177 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqnh\" (UniqueName: \"kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.679218    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:13.679270    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.179253116 +0000 UTC m=+3.143295889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679341    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679426 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679396    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:13.679825 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.679825 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679825 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.679825 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.679758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.682000 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.681971    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.697054 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.697028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqnh\" (UniqueName: \"kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.750835 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.750804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:13.762081 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.762052    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6914d9ae_4112_4a25_9f37_aa04aa12facc.slice/crio-df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263 WatchSource:0}: Error finding container df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263: Status 404 returned error can't find the container with id df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263
Oct 30 18:36:13.773729 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.773707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:13.779897 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.779878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94k6t"
Oct 30 18:36:13.780099 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.780082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:13.782768 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.782745    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf92a54_eb83_4b14_a48c_6a1fea2e11d6.slice/crio-b933d1b04f2c304371cf224b0512e0c94ca7307adf1c895a0e8e669ffca9a31c WatchSource:0}: Error finding container b933d1b04f2c304371cf224b0512e0c94ca7307adf1c895a0e8e669ffca9a31c: Status 404 returned error can't find the container with id b933d1b04f2c304371cf224b0512e0c94ca7307adf1c895a0e8e669ffca9a31c
Oct 30 18:36:13.786224 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.786076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:13.790494 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.790472    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb025a17_a0b0_4f54_9485_b8a85657a54f.slice/crio-ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2 WatchSource:0}: Error finding container ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2: Status 404 returned error can't find the container with id ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2
Oct 30 18:36:13.793275 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.791666    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111c9d09_fd7b_4a3f_99a7_3075467e5e58.slice/crio-19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9 WatchSource:0}: Error finding container 19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9: Status 404 returned error can't find the container with id 19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9
Oct 30 18:36:13.798474 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.798452    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00a8cdd_93b4_459e_b271_03c501a4c5e5.slice/crio-00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b WatchSource:0}: Error finding container 00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b: Status 404 returned error can't find the container with id 00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b
Oct 30 18:36:13.801895 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.801875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:13.811697 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.811674    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda395f190_2d86_410f_81ef_93290e75b3a0.slice/crio-ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0 WatchSource:0}: Error finding container ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0: Status 404 returned error can't find the container with id ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0
Oct 30 18:36:13.821649 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.821628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:13.826372 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.826352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:13.830192 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:13.830165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:13.834926 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.834903    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3132166_1ec1_448e_98c2_b9844cc8ef00.slice/crio-4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839 WatchSource:0}: Error finding container 4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839: Status 404 returned error can't find the container with id 4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839
Oct 30 18:36:13.839638 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:13.839614    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c20dfb_2366_4678_8ffd_9ad49a7774fd.slice/crio-57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5 WatchSource:0}: Error finding container 57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5: Status 404 returned error can't find the container with id 57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5
Oct 30 18:36:14.082038 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.081942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:14.082170 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.082126    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:14.082242 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.082202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.082181893 +0000 UTC m=+4.046224688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:14.182764 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.182728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:14.182905 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.182782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:14.182905 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.182880    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:14.183027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.182909    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:14.183027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.182940    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.182923226 +0000 UTC m=+4.146965998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:14.183027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.182949    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:14.183027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.182964    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:14.183233 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:14.183029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.182997387 +0000 UTC m=+4.147040162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:14.536023 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.535935    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:13 +0000 UTC" deadline="2028-08-02 00:38:48.2309558 +0000 UTC"
Oct 30 18:36:14.536023 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.535998    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15390h2m33.694961442s"
Oct 30 18:36:14.565060 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.564224    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"76487939913b7836b4b3b35bf9597886","Type":"ContainerStarted","Data":"b1f292a956c45e694561402bbfb1e575f7917c17150540fe06453bd052b2e4b4"}
Oct 30 18:36:14.574916 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.574824    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wqk77" event={"ID":"a395f190-2d86-410f-81ef-93290e75b3a0","Type":"ContainerStarted","Data":"ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0"}
Oct 30 18:36:14.581047 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.580930    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96jgv" event={"ID":"f00a8cdd-93b4-459e-b271-03c501a4c5e5","Type":"ContainerStarted","Data":"00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b"}
Oct 30 18:36:14.584082 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.583939    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9"}
Oct 30 18:36:14.595591 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.595418    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8z7tf" event={"ID":"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6","Type":"ContainerStarted","Data":"b933d1b04f2c304371cf224b0512e0c94ca7307adf1c895a0e8e669ffca9a31c"}
Oct 30 18:36:14.601947 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.601835    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc" event={"ID":"8c2d2e58-122c-47f9-8c32-0f73c4e8331c","Type":"ContainerStarted","Data":"ec4c24cbd3a6034c72a2429b90e556466240f90674a9214f3f4c5e175cf162bd"}
Oct 30 18:36:14.606950 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.606864    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerStarted","Data":"57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5"}
Oct 30 18:36:14.610275 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.609818    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5hb6" event={"ID":"d3132166-1ec1-448e-98c2-b9844cc8ef00","Type":"ContainerStarted","Data":"4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839"}
Oct 30 18:36:14.610275 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.610076    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:14.614873 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.614779    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94k6t" event={"ID":"eb025a17-a0b0-4f54-9485-b8a85657a54f","Type":"ContainerStarted","Data":"ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2"}
Oct 30 18:36:14.618503 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:14.618451    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzfd" event={"ID":"6914d9ae-4112-4a25-9f37-aa04aa12facc","Type":"ContainerStarted","Data":"df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263"}
Oct 30 18:36:15.090500 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.089702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:15.090500 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.089920    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:15.090500 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.089987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.089967083 +0000 UTC m=+6.054009858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:15.190386 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.190299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:15.190386 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.190388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:15.190642 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190582    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:15.190642 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190597    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:15.190642 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190608    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:15.190790 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.19064203 +0000 UTC m=+6.154684815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:15.190960 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190944    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:15.191046 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.190982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.190973525 +0000 UTC m=+6.155016310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:15.538119 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.536240    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:13 +0000 UTC" deadline="2028-05-26 08:21:27.086678556 +0000 UTC"
Oct 30 18:36:15.538119 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.536281    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13765h45m11.550402441s"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.538804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.538805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.538942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.539001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.539150    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:15.539872 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:15.539234    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:15.642941 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.642886    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5d63ad00e5a9245d4df7ea9f3aeb927" containerID="95bd95d3c619ce02445d7e62ef41bacb88cf97e80697aa5d204741e37229ec06" exitCode=0
Oct 30 18:36:15.643539 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.643516    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerDied","Data":"95bd95d3c619ce02445d7e62ef41bacb88cf97e80697aa5d204741e37229ec06"}
Oct 30 18:36:15.659634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:15.658688    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal" podStartSLOduration=3.658668886 podStartE2EDuration="3.658668886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:14.580737814 +0000 UTC m=+3.544780610" watchObservedRunningTime="2026-10-30 18:36:15.658668886 +0000 UTC m=+4.622711681"
Oct 30 18:36:16.665349 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:16.665040    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerStarted","Data":"8c0e470012920b7f7435d9d1a4758ff59f69d4a9f46fa815a4b88e1f1b89595c"}
Oct 30 18:36:16.697912 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:16.697855    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" podStartSLOduration=4.697838186 podStartE2EDuration="4.697838186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:16.69760064 +0000 UTC m=+5.661643436" watchObservedRunningTime="2026-10-30 18:36:16.697838186 +0000 UTC m=+5.661880983"
Oct 30 18:36:17.105930 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.105890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:17.106142 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.106110    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:17.106202 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.106181    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.10616136 +0000 UTC m=+10.070204145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.206554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.206627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.206745    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.206809    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.20679188 +0000 UTC m=+10.170834675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.207247    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.207265    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.207278    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.207470 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.207326    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.20730985 +0000 UTC m=+10.171352636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.539348 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.538470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:17.539348 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.538641    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:17.539348 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.539181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:17.539348 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.539207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:17.539348 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.539298    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:17.539767 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:17.539397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:17.673849 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.673793    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="762cfeead1d81e98292ab1b76dbe998f1f34ca357870e7b54d1d99346d34c1e7" exitCode=0
Oct 30 18:36:17.674846 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:17.674821    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"762cfeead1d81e98292ab1b76dbe998f1f34ca357870e7b54d1d99346d34c1e7"}
Oct 30 18:36:18.686335 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:18.686294    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5hb6" event={"ID":"d3132166-1ec1-448e-98c2-b9844cc8ef00","Type":"ContainerStarted","Data":"2a93b3d726c4fe32cd44dfebd151b05522c1c4bfa74f9012b48dbbef6a5db535"}
Oct 30 18:36:19.536113 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.536074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:19.536294 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:19.536230    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:19.536679 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.536661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:19.536784 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:19.536763    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:19.536842 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.536830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:19.536919 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:19.536900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:19.693146 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.693064    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96jgv" event={"ID":"f00a8cdd-93b4-459e-b271-03c501a4c5e5","Type":"ContainerStarted","Data":"f19c27ebf77a236ab10420d2b52a70a4ecd18731671ca42993409ac9aed8a299"}
Oct 30 18:36:19.710435 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.708949    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x5hb6" podStartSLOduration=4.568568007 podStartE2EDuration="8.70892706s" totalImagesPullingTime="4.140359053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:18.705643919 +0000 UTC m=+7.669686709" watchObservedRunningTime="2026-10-30 18:36:19.70892706 +0000 UTC m=+8.672969851"
Oct 30 18:36:19.820165 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:19.820042    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:20.698197 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:20.697983    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc" event={"ID":"8c2d2e58-122c-47f9-8c32-0f73c4e8331c","Type":"ContainerStarted","Data":"f0fd989bcb005e2be88354a08b00d3e734d15b2df6a39cd0ca2a10158484b8d8"}
Oct 30 18:36:21.140485 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.139686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:21.140485 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.139913    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:21.140485 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.139996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.139974475 +0000 UTC m=+18.104017266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:21.240557 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.240509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:21.240753 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.240603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:21.240753 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240675    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:21.240753 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240736    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:21.240753 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.240727509 +0000 UTC m=+18.204770295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:21.240753 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240751    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:21.241034 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240765    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:21.241034 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.240799    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.240789079 +0000 UTC m=+18.204831862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:21.478743 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.478623    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:21.520774 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.520685    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:21.478638834Z","UUID":"bb43ad90-278d-4cf9-b2ea-f84825cc171d","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:21.522559 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.522516    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:21.523067 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.522774    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:21.543305 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.543270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:21.543482 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.543394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:21.543805 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.543786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:21.543932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.543910    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:21.544048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.543977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:21.544138 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:21.544101    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:21.709658 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:21.709312    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc" event={"ID":"8c2d2e58-122c-47f9-8c32-0f73c4e8331c","Type":"ContainerStarted","Data":"3409963931fffa8d278c305441fff94e8ddc32a1ec2aee401187779424ce565a"}
Oct 30 18:36:22.014381 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.014307    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:22.015524 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.015491    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:22.033528 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.033453    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-96jgv" podStartSLOduration=5.976685914 podStartE2EDuration="11.033437216s" totalImagesPullingTime="5.056751302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:19.710414043 +0000 UTC m=+8.674456838" watchObservedRunningTime="2026-10-30 18:36:22.033437216 +0000 UTC m=+10.997480010"
Oct 30 18:36:22.717155 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.716600    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerStarted","Data":"1a999e91bc4aac8d69b22bcf4179ac75b3879194c2db26e9ed08693e70d8e70e"}
Oct 30 18:36:22.719938 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.718766    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94k6t" event={"ID":"eb025a17-a0b0-4f54-9485-b8a85657a54f","Type":"ContainerStarted","Data":"498ee2f4b6b8ae6e1c76aedae9f1864e5f986b82fc1f6cde2a98f6e681a1ca02"}
Oct 30 18:36:22.722306 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.722270    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzfd" event={"ID":"6914d9ae-4112-4a25-9f37-aa04aa12facc","Type":"ContainerStarted","Data":"989b147e57f69f5b84cc4dde28257f006233441cfd4ef3abbb25599d7048f009"}
Oct 30 18:36:22.723140 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.723064    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:22.758384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.758137    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-flzfd" podStartSLOduration=3.26747532 podStartE2EDuration="11.758111957s" totalImagesPullingTime="8.490636637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:22.757455973 +0000 UTC m=+11.721498768" watchObservedRunningTime="2026-10-30 18:36:22.758111957 +0000 UTC m=+11.722154754"
Oct 30 18:36:22.795908 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:22.795785    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-94k6t" podStartSLOduration=3.935926579 podStartE2EDuration="11.795764094s" totalImagesPullingTime="7.859837515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:22.795445943 +0000 UTC m=+11.759488737" watchObservedRunningTime="2026-10-30 18:36:22.795764094 +0000 UTC m=+11.759806890"
Oct 30 18:36:23.539807 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.539718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:23.539987 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.539857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:23.539987 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:23.539866    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:23.540149 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:23.539980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:23.540149 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.540075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:23.540240 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:23.540187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:23.729214 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.729119    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc" event={"ID":"8c2d2e58-122c-47f9-8c32-0f73c4e8331c","Type":"ContainerStarted","Data":"bee1dba1f1f081a1f440fdf296de883b1bd955f8b15116181a6ede603b1320b1"}
Oct 30 18:36:23.732253 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.732218    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="1a999e91bc4aac8d69b22bcf4179ac75b3879194c2db26e9ed08693e70d8e70e" exitCode=0
Oct 30 18:36:23.732457 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.732327    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"1a999e91bc4aac8d69b22bcf4179ac75b3879194c2db26e9ed08693e70d8e70e"}
Oct 30 18:36:23.736609 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.736568    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wqk77" event={"ID":"a395f190-2d86-410f-81ef-93290e75b3a0","Type":"ContainerStarted","Data":"5a15f3d88ab4f3c027dae5e004566050627a6c6eaff7330863d3f75209b4d8e5"}
Oct 30 18:36:23.751843 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.750840    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc" podStartSLOduration=3.424919931 podStartE2EDuration="11.750823233s" totalImagesPullingTime="8.325903302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:23.750694999 +0000 UTC m=+12.714737794" watchObservedRunningTime="2026-10-30 18:36:23.750823233 +0000 UTC m=+12.714866030"
Oct 30 18:36:23.768700 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:23.768630    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wqk77" podStartSLOduration=4.6386538250000005 podStartE2EDuration="12.768612869s" totalImagesPullingTime="8.129959044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:23.768157629 +0000 UTC m=+12.732200422" watchObservedRunningTime="2026-10-30 18:36:23.768612869 +0000 UTC m=+12.732655666"
Oct 30 18:36:24.741515 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:24.741469    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8z7tf" event={"ID":"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6","Type":"ContainerStarted","Data":"b8158725a49aa019e39409e98df6085a10be20149ceade0efc0ca1a07b8ccb01"}
Oct 30 18:36:24.761985 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:24.761914    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8z7tf" podStartSLOduration=2.992410354 podStartE2EDuration="13.761892629s" totalImagesPullingTime="10.769482275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:24.761651665 +0000 UTC m=+13.725694460" watchObservedRunningTime="2026-10-30 18:36:24.761892629 +0000 UTC m=+13.725935425"
Oct 30 18:36:25.536238 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:25.536203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:25.536238 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:25.536248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:25.536516 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:25.536203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:25.536516 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:25.536360    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:25.536516 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:25.536463    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:25.536653 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:25.536559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:25.746914 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:25.746865    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="f8337f71bce69a765455503f007fd60ee0f7b2f58c71a99c8ceecade18b72dc3" exitCode=0
Oct 30 18:36:25.747423 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:25.746938    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"f8337f71bce69a765455503f007fd60ee0f7b2f58c71a99c8ceecade18b72dc3"}
Oct 30 18:36:26.753452 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:26.753353    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="fd7c9661600bda420ff1e3fd71b34bed17d90436b4ee06823ef141445a20ba37" exitCode=0
Oct 30 18:36:26.753452 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:26.753413    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"fd7c9661600bda420ff1e3fd71b34bed17d90436b4ee06823ef141445a20ba37"}
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:27.536274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:27.536294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:27.536429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:27.536437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:27.536552    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:27.536740 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:27.536624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:29.210278 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.210024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:29.210651 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.210189    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:29.210651 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.210447    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:45.210420168 +0000 UTC m=+34.174462953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:29.310824 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.310783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:29.311027 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.310860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:29.311027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.310951    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:29.311027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.310991    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:29.311168 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.311035    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:29.311168 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.311040    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret podName:0eb9d479-2705-4c31-84c1-503e24ae3a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:45.311023214 +0000 UTC m=+34.275065999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret") pod "global-pull-secret-syncer-qw889" (UID: "0eb9d479-2705-4c31-84c1-503e24ae3a0e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:29.311168 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.311051    2579 projected.go:196] Error preparing data for projected volume kube-api-access-b624x for pod openshift-network-diagnostics/network-check-target-lz7bk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:29.311168 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.311111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x podName:b0c7ad43-3851-4baf-8b43-81597fa725b8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:45.311094527 +0000 UTC m=+34.275137300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b624x" (UniqueName: "kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x") pod "network-check-target-lz7bk" (UID: "b0c7ad43-3851-4baf-8b43-81597fa725b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:29.536515 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.536484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:29.536604 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.536491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:29.536723 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.536698    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:29.536771 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.536715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:29.536805 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.536792    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:29.536875 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:29.536855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:29.762893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.762858    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="10e0c2776fbb63710f49853208d5e7f3df6c58448cd28bf301f783acbb413464" exitCode=0
Oct 30 18:36:29.763048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.762909    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"10e0c2776fbb63710f49853208d5e7f3df6c58448cd28bf301f783acbb413464"}
Oct 30 18:36:29.766400 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766374    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"c3b838151a82728dcbe7bb1f957729e04f206c13eb5237a687fd4737d9d494fb"}
Oct 30 18:36:29.766473 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766452    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"4606d92becc083879b4ed0433f98bf3ab075c911e28c14b285d1f69edb3d964a"}
Oct 30 18:36:29.766503 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766484    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"94b426d30e8294ea1d0801a3294d1a6f3d3953278c548ca88bbcaeb2db47fd7a"}
Oct 30 18:36:29.766540 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"5cfd63353f1f0c7b13f2132d64c33ade9a04e901b76dc43d70bc3ef941510383"}
Oct 30 18:36:29.766568 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766542    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"2a1abee3904d42171bb716f112ad72c8e4416afb205726260e178b400bdf4548"}
Oct 30 18:36:29.766596 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:29.766570    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"94b813a8f7a754bae81fd159b6620d088daea0b662d7a3d2ff48b191c058439f"}
Oct 30 18:36:30.773162 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:30.773119    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="bd4d3ed5d9aa0e4d5a435abe1f17b3f6611d84cca74f2a8aa0c95711985cf9fa" exitCode=0
Oct 30 18:36:30.773665 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:30.773199    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerDied","Data":"bd4d3ed5d9aa0e4d5a435abe1f17b3f6611d84cca74f2a8aa0c95711985cf9fa"}
Oct 30 18:36:31.536692 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.536660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:31.536861 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.536782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:31.536861 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:31.536793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:31.536933 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:31.536871    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:31.536933 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.536882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:31.536990 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:31.536943    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:31.779686 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.779655    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"75392e72ad179105da9a92eace6f37bd2a094c070820cce266db28d003a0ee19"}
Oct 30 18:36:31.783267 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.783235    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" event={"ID":"49c20dfb-2366-4678-8ffd-9ad49a7774fd","Type":"ContainerStarted","Data":"8e1b3a7ae59e04f03aa1f835f64aa9d2fb9ef9f7f837d0bf4b7111ba10c451be"}
Oct 30 18:36:31.808736 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:31.808642    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bxfx7" podStartSLOduration=9.150077825 podStartE2EDuration="20.808625501s" totalImagesPullingTime="11.658547676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:31.807736463 +0000 UTC m=+20.771779288" watchObservedRunningTime="2026-10-30 18:36:31.808625501 +0000 UTC m=+20.772668294"
Oct 30 18:36:33.535434 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:33.535399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:33.535893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:33.535398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:33.535893 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:33.535564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:33.535893 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:33.535581    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:33.535893 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:33.535683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:33.535893 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:33.535778    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:34.798245 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:34.798201    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" event={"ID":"111c9d09-fd7b-4a3f-99a7-3075467e5e58","Type":"ContainerStarted","Data":"b82e503678abf43a2d27d185ebf7bf437be0227ea1129758a13e5058bfada506"}
Oct 30 18:36:34.798905 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:34.798494    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:34.798905 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:34.798535    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:34.813341 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:34.813321    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:34.830136 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:34.830091    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l" podStartSLOduration=8.399469364 podStartE2EDuration="23.830079558s" totalImagesPullingTime="15.430610194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:34.829638275 +0000 UTC m=+23.793681069" watchObservedRunningTime="2026-10-30 18:36:34.830079558 +0000 UTC m=+23.794122351"
Oct 30 18:36:35.520875 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.520842    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lz7bk"]
Oct 30 18:36:35.521165 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.520975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:35.521165 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:35.521081    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:35.524100 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.524070    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qw889"]
Oct 30 18:36:35.524219 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.524195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:35.524318 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:35.524295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:35.524671 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.524651    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfglz"]
Oct 30 18:36:35.524772 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.524759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:35.524881 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:35.524861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:35.800858 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.800792    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:35.815565 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:35.815540    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:37.535960 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:37.535767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:37.536417 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:37.535777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:37.536417 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:37.536082    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:37.536417 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:37.535810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:37.536417 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:37.536162    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:37.536417 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:37.536182    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:39.536074 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:39.536045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:39.536074 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:39.536059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:39.536535 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:39.536159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:39.536535 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:39.536159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:39.536535 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:39.536262    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:39.536535 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:39.536359    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:41.536743 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:41.536707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:41.537216 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:41.536861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:41.537216 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:41.536882    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw889" podUID="0eb9d479-2705-4c31-84c1-503e24ae3a0e"
Oct 30 18:36:41.537216 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:41.536953    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lz7bk" podUID="b0c7ad43-3851-4baf-8b43-81597fa725b8"
Oct 30 18:36:41.537216 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:41.536965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:41.537216 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:41.537091    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfglz" podUID="f5c18994-379c-4961-beac-4a01d127cef0"
Oct 30 18:36:42.377003 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.376935    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeReady"
Oct 30 18:36:42.377169 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.377073    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:42.453495 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.453459    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"]
Oct 30 18:36:42.503438 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.503227    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9q9f"]
Oct 30 18:36:42.503438 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.503375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.508757 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.508728    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:42.508967 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.508951    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lfspj\""
Oct 30 18:36:42.509121 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.509058    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:42.535586 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.535556    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7v9n7"]
Oct 30 18:36:42.535741 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.535663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:42.555819 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.555784    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64w7x"]
Oct 30 18:36:42.556179 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.555882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:42.580931 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.580902    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"]
Oct 30 18:36:42.581099 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.581046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:42.598455 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.598430    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fs9gh"]
Oct 30 18:36:42.598570 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.598549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.607199 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.607328 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.607402 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607357    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.607402 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607376    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.607523 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607447    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.607570 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607543    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.607747 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.607732    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:42.608563 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.608545    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqtqh\""
Oct 30 18:36:42.608645 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.608585    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:42.612516 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.612500    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77bc686d65-bdnkc"]
Oct 30 18:36:42.612623 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.612609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.626119 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.626098    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-spj7q"]
Oct 30 18:36:42.626231 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.626179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.648385 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.648355    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"]
Oct 30 18:36:42.648504 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.648470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.663411 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.663381    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"]
Oct 30 18:36:42.663535 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.663487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.677709 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.677682    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"]
Oct 30 18:36:42.677833 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.677768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:42.696109 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.696076    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:36:42.696261 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.696248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.707639 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.707748 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkcl\" (UniqueName: \"kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl\") pod \"kas-connection-checker-685895c7c9-7v9n7\" (UID: \"2cb00238-6e40-41f7-b8c4-05f7ef60e913\") " pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:42.707748 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.707748 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.707859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707746    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:42.707859 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.707787    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:42.707859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707806    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.707859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.707966 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.707865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.207845065 +0000 UTC m=+32.171887853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:42.707966 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707934    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708049 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.707979    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708049 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708042    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.708108 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.708143 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708108    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.708143 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.708203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708162    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:42.708203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.708203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708177    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708203 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708162    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708201    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4nw\" (UniqueName: \"kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708256    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708278    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708291    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.708313 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.708492 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708314    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.708492 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvh28\" (UniqueName: \"kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28\") pod \"kas-connection-checker-685895c7c9-64w7x\" (UID: \"e844bc94-4bbe-410b-9cb6-807b45c555df\") " pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:42.708492 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.708492 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75gw\" (UniqueName: \"kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw\") pod \"kas-connection-checker-685895c7c9-j9q9f\" (UID: \"28992bb6-3816-46b7-af81-3b1a3932432c\") " pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:42.708492 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708574    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nzpfq\""
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708581    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708585    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:42.708630 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh68w\" (UniqueName: \"kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.708790 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.708790 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708651    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:42.708790 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwm9z\" (UniqueName: \"kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.708790 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708684    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-khvcj\""
Oct 30 18:36:42.708790 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708678    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:42.708942 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708850    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:42.708942 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.708892    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v8cvv\""
Oct 30 18:36:42.709136 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709118    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:42.709136 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709128    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.709251 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709135    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:42.709251 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709167    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.709251 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709189    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5twcv\""
Oct 30 18:36:42.709251 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709248    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:42.709367 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709249    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-82qzs\""
Oct 30 18:36:42.709491 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:42.709941 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.709929    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b8djd\""
Oct 30 18:36:42.714274 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.714252    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:42.714401 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.714362    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:42.724586 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.724563    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"]
Oct 30 18:36:42.724756 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.724738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.745355 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.745318    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:36:42.745470 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.745440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:42.766060 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.766031    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"]
Oct 30 18:36:42.766190 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.766111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.787911 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787878    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9q9f"]
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787923    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"]
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787941    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7v9n7"]
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787955    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64w7x"]
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787969    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"]
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:42.788066 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.787987    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5cdsk"]
Oct 30 18:36:42.802156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.802128    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rc44q"]
Oct 30 18:36:42.802265 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.802236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:42.808663 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808642    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:42.808744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808658    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.808744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808719    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:42.808744 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808740    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.808836 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808791    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.808836 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808824    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:42.808964 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808947    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.808998 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808978    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:42.808998 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.808975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.809084 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.809084 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.809147 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.809147 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809098    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:42.809147 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809147 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.809260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.809260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.809260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809216    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x9qkh\""
Oct 30 18:36:42.809260 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809231    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bskc2\""
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809276    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.809298    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809312    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qfk8m\""
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809324    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wskq9\""
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.809353    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.309340496 +0000 UTC m=+32.273383269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809425 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809417    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwm9z\" (UniqueName: \"kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.809468    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809541    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.809579    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.309561442 +0000 UTC m=+32.273604226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh68w\" (UniqueName: \"kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkcl\" (UniqueName: \"kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl\") pod \"kas-connection-checker-685895c7c9-7v9n7\" (UID: \"2cb00238-6e40-41f7-b8c4-05f7ef60e913\") " pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cwn\" (UniqueName: \"kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.809751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblvj\" (UniqueName: \"kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj\") pod \"volume-data-source-validator-58c9f5f46d-qc5dv\" (UID: \"200dacf8-0b68-43ea-87ec-e5f67becf695\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qg6m\" (UniqueName: \"kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.809839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4nw\" (UniqueName: \"kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvh28\" (UniqueName: \"kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28\") pod \"kas-connection-checker-685895c7c9-64w7x\" (UID: \"e844bc94-4bbe-410b-9cb6-807b45c555df\") " pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75gw\" (UniqueName: \"kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw\") pod \"kas-connection-checker-685895c7c9-j9q9f\" (UID: \"28992bb6-3816-46b7-af81-3b1a3932432c\") " pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.810413 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skd9s\" (UniqueName: \"kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.811088 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810448    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.811088 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.810509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.811198 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.811154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.811307 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.811259    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:42.811726 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.811706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.812069 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.812049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.812174 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.812159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.312144606 +0000 UTC m=+32.276187398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:42.813163 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.813140    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.813276 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.813205    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.813341 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.813322    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.819491 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819464    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819501    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819519    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819534    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-spj7q"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819554    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77bc686d65-bdnkc"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819575    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819595    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc44q"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819620    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cdsk"]
Oct 30 18:36:42.819647 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819639    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fs9gh"]
Oct 30 18:36:42.820048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819660    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"]
Oct 30 18:36:42.820048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819683    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:36:42.820048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819703    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"]
Oct 30 18:36:42.820048 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.819841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:42.821156 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.821135    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwm9z\" (UniqueName: \"kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:42.826083 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.826054    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkcl\" (UniqueName: \"kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl\") pod \"kas-connection-checker-685895c7c9-7v9n7\" (UID: \"2cb00238-6e40-41f7-b8c4-05f7ef60e913\") " pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:42.826167 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.826132    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh68w\" (UniqueName: \"kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.827532 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.827505    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvh28\" (UniqueName: \"kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28\") pod \"kas-connection-checker-685895c7c9-64w7x\" (UID: \"e844bc94-4bbe-410b-9cb6-807b45c555df\") " pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:42.827532 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.827523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4nw\" (UniqueName: \"kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:42.834889 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.834859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75gw\" (UniqueName: \"kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw\") pod \"kas-connection-checker-685895c7c9-j9q9f\" (UID: \"28992bb6-3816-46b7-af81-3b1a3932432c\") " pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:42.857877 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.857853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:42.889829 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.889763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:42.906962 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.906928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:42.909489 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.909462    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxh65\""
Oct 30 18:36:42.909489 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.909462    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:42.909682 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.909517    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:42.910934 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.910907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.911086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.910942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.911086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.910961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.911086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.910980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skd9s\" (UniqueName: \"kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.911086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.911286 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.911286 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.911408 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.911472 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpc5\" (UniqueName: \"kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lpp\" (UniqueName: \"kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp\") pod \"network-check-source-6f78d59f99-8tvw8\" (UID: \"cb0c4450-cc6e-4124-9177-af902ae797ba\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnrh\" (UniqueName: \"kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.911997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlsr4\" (UniqueName: \"kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:42.912441 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cwn\" (UniqueName: \"kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblvj\" (UniqueName: \"kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj\") pod \"volume-data-source-validator-58c9f5f46d-qc5dv\" (UID: \"200dacf8-0b68-43ea-87ec-e5f67becf695\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qg6m\" (UniqueName: \"kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912320    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:42.913235 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.912697    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.912892    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.912912    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.912969    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.412950965 +0000 UTC m=+32.376993739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.913116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.913285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.913339    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:42.913390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.413375601 +0000 UTC m=+32.377418375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:42.913932 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.913516    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.914361 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.914296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.914729 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.914700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.915367 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.915347    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.915519 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.915501    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.923455 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.923425    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.923625 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.923608    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:42.924900 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.924865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skd9s\" (UniqueName: \"kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:42.925625 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.925591    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cwn\" (UniqueName: \"kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.925809 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.925792    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblvj\" (UniqueName: \"kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj\") pod \"volume-data-source-validator-58c9f5f46d-qc5dv\" (UID: \"200dacf8-0b68-43ea-87ec-e5f67becf695\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:42.926059 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.925998    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qg6m\" (UniqueName: \"kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:42.935869 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.935840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:42.974495 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.974461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:42.998866 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:42.998830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlsr4\" (UniqueName: \"kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.023634 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpc5\" (UniqueName: \"kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lpp\" (UniqueName: \"kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp\") pod \"network-check-source-6f78d59f99-8tvw8\" (UID: \"cb0c4450-cc6e-4124-9177-af902ae797ba\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnrh\" (UniqueName: \"kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.013721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.014708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.015133    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.015152    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.015215    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.515193122 +0000 UTC m=+32.479235908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.015653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.015768    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.015816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.515800368 +0000 UTC m=+32.479843157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.016253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.017359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.018313    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.018697    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.018980    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.019808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:43.024945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.020559    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.025952 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.022109    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.030823 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.030790    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpc5\" (UniqueName: \"kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.031473 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.031415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlsr4\" (UniqueName: \"kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:43.034610 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.033990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnrh\" (UniqueName: \"kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.037536 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.037508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.038094 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.038063    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.038575 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.038555    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lpp\" (UniqueName: \"kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp\") pod \"network-check-source-6f78d59f99-8tvw8\" (UID: \"cb0c4450-cc6e-4124-9177-af902ae797ba\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:43.081104 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.078458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:43.121110 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.120902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:43.195257 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.195204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:43.214998 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.214961    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9q9f"]
Oct 30 18:36:43.215571 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.215546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:43.215724 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.215709    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:43.215803 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.215766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.215753471 +0000 UTC m=+33.179796243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:43.219243 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.219216    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28992bb6_3816_46b7_af81_3b1a3932432c.slice/crio-8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08 WatchSource:0}: Error finding container 8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08: Status 404 returned error can't find the container with id 8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08
Oct 30 18:36:43.222672 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.222645    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7v9n7"]
Oct 30 18:36:43.229930 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.229901    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb00238_6e40_41f7_b8c4_05f7ef60e913.slice/crio-42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8 WatchSource:0}: Error finding container 42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8: Status 404 returned error can't find the container with id 42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8
Oct 30 18:36:43.246379 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.246284    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64w7x"]
Oct 30 18:36:43.251211 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.251155    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode844bc94_4bbe_410b_9cb6_807b45c555df.slice/crio-4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648 WatchSource:0}: Error finding container 4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648: Status 404 returned error can't find the container with id 4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648
Oct 30 18:36:43.277295 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.277265    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fs9gh"]
Oct 30 18:36:43.281027 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.280976    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1e8569_d080_44ba_8c4b_b5ae19eaa04c.slice/crio-e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495 WatchSource:0}: Error finding container e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495: Status 404 returned error can't find the container with id e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495
Oct 30 18:36:43.317753 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.317713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:43.317929 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.317782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:43.317929 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.317812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:43.318053 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.317971    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:43.318053 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.318048    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.31803066 +0000 UTC m=+33.282073445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:43.318504 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.318482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.318463346 +0000 UTC m=+33.282506140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:43.319956 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.318570    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:43.319956 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.318612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.318597928 +0000 UTC m=+33.282640714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:43.321428 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.321320    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"]
Oct 30 18:36:43.321428 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.321380    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-spj7q"]
Oct 30 18:36:43.345343 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.345299    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"]
Oct 30 18:36:43.398454 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.398158    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"]
Oct 30 18:36:43.404076 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.404041    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f16f0_f04f_4ea8_8e51_a1d2ca19858a.slice/crio-8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0 WatchSource:0}: Error finding container 8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0: Status 404 returned error can't find the container with id 8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0
Oct 30 18:36:43.418739 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.418712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:43.418840 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.418766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:43.418893 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.418860    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:43.418933 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.418916    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.418901625 +0000 UTC m=+33.382944414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:43.418971 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.418861    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:43.418971 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.418959    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:43.419060 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.419020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.418994013 +0000 UTC m=+33.383036803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:43.424431 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.424406    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"]
Oct 30 18:36:43.427454 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.427432    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c4450_cc6e_4124_9177_af902ae797ba.slice/crio-748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41 WatchSource:0}: Error finding container 748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41: Status 404 returned error can't find the container with id 748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41
Oct 30 18:36:43.515103 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.514978    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cdsk"]
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.519279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.519348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.519604    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.519667    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.519648918 +0000 UTC m=+33.483691706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.520122    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.520138    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:43.520185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:44.52016996 +0000 UTC m=+33.484212749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:43.520272 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:43.520250    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09bbce7_7e03_435e_992a_c1d89acdbe5a.slice/crio-6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1 WatchSource:0}: Error finding container 6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1: Status 404 returned error can't find the container with id 6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1
Oct 30 18:36:43.540509 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.540480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:43.540668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.540518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:43.540668 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.540478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:43.544783 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.544759    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:43.545561 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.545545    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-829tl\""
Oct 30 18:36:43.546493 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.546472    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-75g8c\""
Oct 30 18:36:43.546631 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.546615    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:43.546726 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.546711    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2njn\""
Oct 30 18:36:43.820281 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.820247    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8" event={"ID":"cb0c4450-cc6e-4124-9177-af902ae797ba","Type":"ContainerStarted","Data":"748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41"}
Oct 30 18:36:43.821829 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.821796    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d" event={"ID":"a9506e6a-0145-4e49-b3f1-acd79fc488f5","Type":"ContainerStarted","Data":"62ba3faaf9f0c3bd94653b4af2cc0d4679a3704e78ccfba567d66edb6fa82daa"}
Oct 30 18:36:43.823557 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.823524    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-64w7x" event={"ID":"e844bc94-4bbe-410b-9cb6-807b45c555df","Type":"ContainerStarted","Data":"b84d02406e3963b8128e3ff0d39b00d7c8253d8654c4713cbeff3e60400dca34"}
Oct 30 18:36:43.823664 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.823572    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-64w7x" event={"ID":"e844bc94-4bbe-410b-9cb6-807b45c555df","Type":"ContainerStarted","Data":"4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648"}
Oct 30 18:36:43.825073 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.825044    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cdsk" event={"ID":"a09bbce7-7e03-435e-992a-c1d89acdbe5a","Type":"ContainerStarted","Data":"6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1"}
Oct 30 18:36:43.826389 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.826363    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv" event={"ID":"200dacf8-0b68-43ea-87ec-e5f67becf695","Type":"ContainerStarted","Data":"7f97130927c38ccb9e0355ffad92ec9e6ad9fd925751f47ee344b52d434e7f91"}
Oct 30 18:36:43.828483 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.828456    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fs9gh" event={"ID":"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c","Type":"ContainerStarted","Data":"e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495"}
Oct 30 18:36:43.829977 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.829935    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn" event={"ID":"708f16f0-f04f-4ea8-8e51-a1d2ca19858a","Type":"ContainerStarted","Data":"8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0"}
Oct 30 18:36:43.831728 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.831702    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" event={"ID":"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03","Type":"ContainerStarted","Data":"a9e585e753d712a584ae7d5046878da28274cfe15bce41be2a25b0bcda951e39"}
Oct 30 18:36:43.833612 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.833578    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7v9n7" event={"ID":"2cb00238-6e40-41f7-b8c4-05f7ef60e913","Type":"ContainerStarted","Data":"a07e601885427debec07bb80a7485b87401d6631a9d2166a4153e5bc84644863"}
Oct 30 18:36:43.833763 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.833641    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7v9n7" event={"ID":"2cb00238-6e40-41f7-b8c4-05f7ef60e913","Type":"ContainerStarted","Data":"42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8"}
Oct 30 18:36:43.835907 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.835879    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j9q9f" event={"ID":"28992bb6-3816-46b7-af81-3b1a3932432c","Type":"ContainerStarted","Data":"dc63c839e9d1d188757a41d5f394c66691b1a792e6e9647d4d4a1a51f8582bf2"}
Oct 30 18:36:43.836053 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.835928    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j9q9f" event={"ID":"28992bb6-3816-46b7-af81-3b1a3932432c","Type":"ContainerStarted","Data":"8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08"}
Oct 30 18:36:43.844756 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.844709    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-64w7x" podStartSLOduration=303.844694592 podStartE2EDuration="5m3.844694592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.844126427 +0000 UTC m=+32.808169232" watchObservedRunningTime="2026-10-30 18:36:43.844694592 +0000 UTC m=+32.808737388"
Oct 30 18:36:43.869105 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:43.869040    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7v9n7" podStartSLOduration=303.869021588 podStartE2EDuration="5m3.869021588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.867416163 +0000 UTC m=+32.831458958" watchObservedRunningTime="2026-10-30 18:36:43.869021588 +0000 UTC m=+32.833064380"
Oct 30 18:36:44.227138 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.227089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:44.227340 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.227280    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:44.227398 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.227363    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.227340514 +0000 UTC m=+35.191383288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:44.328984 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.328946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:44.329182 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.329027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:44.329182 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.329057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:44.329338 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.329321    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:44.329393 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.329385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.329367519 +0000 UTC m=+35.293410292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:44.329614 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.329569    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:44.329792 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.329748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.329446655 +0000 UTC m=+35.293489441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:44.329792 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.329775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.32976454 +0000 UTC m=+35.293807320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.430046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.430247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.430410    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.430425    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.430486    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.430465558 +0000 UTC m=+35.394508331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.430907    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:44.431027 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.430962    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.430946332 +0000 UTC m=+35.394989110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.531770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:44.531851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.532271    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.532335    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.532317052 +0000 UTC m=+35.496359844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.532789    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.532805    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:44.533043 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:44.532885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.532833869 +0000 UTC m=+35.496876656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:45.241205 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.240649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:45.241205 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:45.240903    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:45.241205 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:45.240969    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:17.240950487 +0000 UTC m=+66.204993273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : secret "metrics-daemon-secret" not found
Oct 30 18:36:45.341512 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.341477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:45.341667 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.341558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:45.357036 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.346537    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:45.357036 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.348516    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:45.366983 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.366918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:45.395037 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.394935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:45.796172 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.794182    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j9q9f" podStartSLOduration=305.794162872 podStartE2EDuration="5m5.794162872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.896944194 +0000 UTC m=+32.860986989" watchObservedRunningTime="2026-10-30 18:36:45.794162872 +0000 UTC m=+34.758205666"
Oct 30 18:36:45.796172 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.794520    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qw889"]
Oct 30 18:36:45.796172 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.796107    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lz7bk"]
Oct 30 18:36:45.798828 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:45.798182    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c7ad43_3851_4baf_8b43_81597fa725b8.slice/crio-d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986 WatchSource:0}: Error finding container d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986: Status 404 returned error can't find the container with id d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986
Oct 30 18:36:45.850572 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.850109    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qw889" event={"ID":"0eb9d479-2705-4c31-84c1-503e24ae3a0e","Type":"ContainerStarted","Data":"adf53e3927bab969973feb3684fed133230756bd4fc55e5dab9d713080f08a17"}
Oct 30 18:36:45.854861 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.854081    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv" event={"ID":"200dacf8-0b68-43ea-87ec-e5f67becf695","Type":"ContainerStarted","Data":"f2257c071de63dee08abf1d8f7f21a1106928773d1764da2fd27a72b8910aa1f"}
Oct 30 18:36:45.856895 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.856824    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lz7bk" event={"ID":"b0c7ad43-3851-4baf-8b43-81597fa725b8","Type":"ContainerStarted","Data":"d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986"}
Oct 30 18:36:45.876995 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:45.876370    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv" podStartSLOduration=263.922962666 podStartE2EDuration="4m25.876346653s" totalImagesPullingTime="1.953383987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:45.875209546 +0000 UTC m=+34.839252340" watchObservedRunningTime="2026-10-30 18:36:45.876346653 +0000 UTC m=+34.840389461"
Oct 30 18:36:46.254480 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.254442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:46.254943 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.254676    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:46.254943 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.254745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.254724755 +0000 UTC m=+39.218767531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.356429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.356478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.356620    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.356644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.356651    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.356699    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.356679159 +0000 UTC m=+39.320721933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.356722    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.356710614 +0000 UTC m=+39.320753391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:46.356743 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.356737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.35672858 +0000 UTC m=+39.320771386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:46.457415 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.457374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:46.457611 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.457463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:46.457611 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.457606    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:46.457782 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.457669    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.457651559 +0000 UTC m=+39.421694338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:46.458203 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.458075    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:46.458203 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.458094    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:46.458203 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.458137    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.458122277 +0000 UTC m=+39.422165054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:46.558828 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.558369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:46.558828 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:46.558443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:46.558828 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.558630    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:46.559829 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.559360    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.559336129 +0000 UTC m=+39.523378916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:46.559829 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.558728    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:46.559829 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.559717    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:46.559829 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:46.559766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.559752197 +0000 UTC m=+39.523794974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:48.896960 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.896864    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fs9gh" event={"ID":"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c","Type":"ContainerStarted","Data":"41ce3c148b328acf8d23045b751d824337b8aa2da224533e392b7adb5ceb2faf"}
Oct 30 18:36:48.899936 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.899895    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn" event={"ID":"708f16f0-f04f-4ea8-8e51-a1d2ca19858a","Type":"ContainerStarted","Data":"8107b3c3fc53a8331e0950b4c6f1fe8e8ebc75f4221e94f766b227a7627161c7"}
Oct 30 18:36:48.905945 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.905916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/0.log"
Oct 30 18:36:48.906086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.905953    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" containerID="987d46e6e4939201f0d17688bf336a403020115a1591c73a3c1f7ce2a3dc1b2d" exitCode=255
Oct 30 18:36:48.906086 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.906060    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" event={"ID":"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03","Type":"ContainerDied","Data":"987d46e6e4939201f0d17688bf336a403020115a1591c73a3c1f7ce2a3dc1b2d"}
Oct 30 18:36:48.906395 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.906366    2579 scope.go:122] "RemoveContainer" containerID="987d46e6e4939201f0d17688bf336a403020115a1591c73a3c1f7ce2a3dc1b2d"
Oct 30 18:36:48.909279 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.909250    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d" event={"ID":"a9506e6a-0145-4e49-b3f1-acd79fc488f5","Type":"ContainerStarted","Data":"9ed675487e070ba2ad2987c8b7f2ab44d69fe921cedd18f0fbe27d675ce52a52"}
Oct 30 18:36:48.920354 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.919376    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fs9gh" podStartSLOduration=263.953667219 podStartE2EDuration="4m28.919361871s" totalImagesPullingTime="4.965694652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:48.918698212 +0000 UTC m=+37.882741006" watchObservedRunningTime="2026-10-30 18:36:48.919361871 +0000 UTC m=+37.883404666"
Oct 30 18:36:48.984751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:48.983619    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d" podStartSLOduration=264.031135481 podStartE2EDuration="4m28.983598969s" totalImagesPullingTime="4.952463488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:48.982091209 +0000 UTC m=+37.946134004" watchObservedRunningTime="2026-10-30 18:36:48.983598969 +0000 UTC m=+37.947641764"
Oct 30 18:36:49.004314 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.003301    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4x55m"]
Oct 30 18:36:49.024755 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.024704    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn" podStartSLOduration=263.807431965 podStartE2EDuration="4m29.024688022s" totalImagesPullingTime="5.217256057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.014147905 +0000 UTC m=+37.978190700" watchObservedRunningTime="2026-10-30 18:36:49.024688022 +0000 UTC m=+37.988730851"
Oct 30 18:36:49.037060 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.036962    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4x55m"]
Oct 30 18:36:49.037312 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.037154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.041369 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.041344    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:49.041531 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.041434    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r4rtj\""
Oct 30 18:36:49.041609 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.041463    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:49.200331 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.200294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.200502 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.200408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.200502 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.200443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.200583 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.200539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.200618 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.200598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7f87\" (UniqueName: \"kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301237 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301410 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301473 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.301398    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.301524 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.301476    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:49.801455919 +0000 UTC m=+38.765498692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.301751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301751 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301859 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7f87\" (UniqueName: \"kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.301954 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.301867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.302046 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.302000    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:49.302115 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.302082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:49.802068246 +0000 UTC m=+38.766111021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:49.318866 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.318810    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7f87\" (UniqueName: \"kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.712862 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.712820    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"]
Oct 30 18:36:49.723431 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.723403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:49.731757 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.731656    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"]
Oct 30 18:36:49.738862 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.738810    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-82wlx\""
Oct 30 18:36:49.797762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.797730    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:49.797762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.797731    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:49.809591 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.809559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.809761 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.809628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:49.809761 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.809684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfxb\" (UniqueName: \"kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb\") pod \"migrator-66df5d496b-8x557\" (UID: \"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:49.809761 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.809713    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.809761 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.809715    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:49.809976 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.809781    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.80976127 +0000 UTC m=+39.773804058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.809976 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.809831    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.80981145 +0000 UTC m=+39.773854241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:49.910657 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.910414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfxb\" (UniqueName: \"kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb\") pod \"migrator-66df5d496b-8x557\" (UID: \"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:49.918105 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.918062    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cdsk" event={"ID":"a09bbce7-7e03-435e-992a-c1d89acdbe5a","Type":"ContainerStarted","Data":"269bc4e5c3718c27f9899e1498cc4bbf58b55b063ff22a3c279abd53d5419169"}
Oct 30 18:36:49.921474 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.921442    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:49.921962 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.921944    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/0.log"
Oct 30 18:36:49.922100 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.921984    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d" exitCode=255
Oct 30 18:36:49.922163 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.922094    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" event={"ID":"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03","Type":"ContainerDied","Data":"f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"}
Oct 30 18:36:49.922163 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.922142    2579 scope.go:122] "RemoveContainer" containerID="987d46e6e4939201f0d17688bf336a403020115a1591c73a3c1f7ce2a3dc1b2d"
Oct 30 18:36:49.922384 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.922364    2579 scope.go:122] "RemoveContainer" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"
Oct 30 18:36:49.922611 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:49.922584    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:36:49.927045 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.926966    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfxb\" (UniqueName: \"kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb\") pod \"migrator-66df5d496b-8x557\" (UID: \"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:49.965856 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:49.965757    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5cdsk" podStartSLOduration=2.148376839 podStartE2EDuration="7.965733739s" totalImagesPullingTime="5.8173569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.940159149 +0000 UTC m=+38.904201943" watchObservedRunningTime="2026-10-30 18:36:49.965733739 +0000 UTC m=+38.929776533"
Oct 30 18:36:50.049170 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.049134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:50.314778 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.314699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:50.315497 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.315034    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:50.315497 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.315108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.315085918 +0000 UTC m=+47.279128710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:50.378695 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.378609    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"]
Oct 30 18:36:50.384286 ip-10-0-134-34 kubenswrapper[2579]: W1030 18:36:50.384258    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91a9ddf_06c5_45fa_8d2f_a565fa2f52fb.slice/crio-f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460 WatchSource:0}: Error finding container f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460: Status 404 returned error can't find the container with id f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460
Oct 30 18:36:50.415705 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.415671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:50.415872 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.415730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:50.415872 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.415826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:50.415872 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.415850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.415828043 +0000 UTC m=+47.379870854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:50.415872 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.415855    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:50.416105 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.415902    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:50.416105 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.415914    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.41589749 +0000 UTC m=+47.379940263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:50.416105 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.415945    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.415931799 +0000 UTC m=+47.379974585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.517526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.517707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.517863    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.517876    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.517929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.517912048 +0000 UTC m=+47.481954836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.518442    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:50.518519 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.518488    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.51847773 +0000 UTC m=+47.482520516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:50.619303 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.619217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:50.619303 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.619290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:50.619487 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.619395    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:50.619487 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.619427    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:50.619487 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.619401    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:50.619487 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.619481    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.619467253 +0000 UTC m=+47.583510025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:50.619611 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.619499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.619490978 +0000 UTC m=+47.583533751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:50.821269 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.821226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:50.821471 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.821320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:50.821471 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.821405    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:50.821471 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.821447    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:50.821650 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.821486    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:52.821465981 +0000 UTC m=+41.785508753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:50.821650 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.821510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:52.821493534 +0000 UTC m=+41.785536321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:50.934392 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.934356    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557" event={"ID":"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb","Type":"ContainerStarted","Data":"f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460"}
Oct 30 18:36:50.936564 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.936537    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:50.937246 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:50.937223    2579 scope.go:122] "RemoveContainer" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"
Oct 30 18:36:50.937431 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:50.937412    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:36:51.941698 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:51.941652    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lz7bk" event={"ID":"b0c7ad43-3851-4baf-8b43-81597fa725b8","Type":"ContainerStarted","Data":"facf9d183bbd86fab87d9304f51699752259388fe5c81cf49ae796454f0ee3b6"}
Oct 30 18:36:51.942114 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:51.941759    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:51.943762 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:51.943739    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8" event={"ID":"cb0c4450-cc6e-4124-9177-af902ae797ba","Type":"ContainerStarted","Data":"c3f6881a67bcddd398a34104c91e8b62f02c5c4b17f87b1e3baaf2b38c37404c"}
Oct 30 18:36:51.971982 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:51.971921    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lz7bk" podStartSLOduration=35.430603853 podStartE2EDuration="40.971902599s" totalImagesPullingTime="5.541298746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:51.970653769 +0000 UTC m=+40.934696564" watchObservedRunningTime="2026-10-30 18:36:51.971902599 +0000 UTC m=+40.935945440"
Oct 30 18:36:52.012680 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.012623    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8" podStartSLOduration=263.595723373 podStartE2EDuration="4m31.012605877s" totalImagesPullingTime="7.416882504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:52.011362099 +0000 UTC m=+40.975404894" watchObservedRunningTime="2026-10-30 18:36:52.012605877 +0000 UTC m=+40.976648671"
Oct 30 18:36:52.840124 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.840087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:52.840278 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.840231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:52.840278 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:52.840233    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:52.840345 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:52.840294    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.84028004 +0000 UTC m=+45.804322812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:52.840345 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:52.840301    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:52.840345 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:52.840342    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.840330963 +0000 UTC m=+45.804373734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:52.949194 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.949154    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557" event={"ID":"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb","Type":"ContainerStarted","Data":"ada96efa4e2dc2ad148e11f517a73e4f329c1bb7b9809261ba32cd0f3f844140"}
Oct 30 18:36:52.965768 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.965712    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557" podStartSLOduration=2.251908304 podStartE2EDuration="3.965694983s" totalImagesPullingTime="1.713786679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:52.965486402 +0000 UTC m=+41.929529197" watchObservedRunningTime="2026-10-30 18:36:52.965694983 +0000 UTC m=+41.929737778"
Oct 30 18:36:52.975525 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.975504    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:52.975621 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.975567    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:52.975842 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:52.975830    2579 scope.go:122] "RemoveContainer" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"
Oct 30 18:36:52.976018 ip-10-0-134-34 kubenswrapper[2579]: E1030 18:36:52.975990    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:36:53.334669 ip-10-0-134-34 kubenswrapper[2579]: I1030 18:36:53.334594    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:53.334755 ip-10-0-134-34 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:36:53.350348 ip-10-0-134-34 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:36:53.350565 ip-10-0-134-34 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:36:53.350625 ip-10-0-134-34 systemd[1]: kubelet.service: Consumed 3.308s CPU time.
Oct 30 18:36:53.361608 ip-10-0-134-34 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:53.452136 ip-10-0-134-34 kubenswrapper[4942]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:53.452136 ip-10-0-134-34 kubenswrapper[4942]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:53.452136 ip-10-0-134-34 kubenswrapper[4942]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:53.452136 ip-10-0-134-34 kubenswrapper[4942]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:53.454329 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454235    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.454329 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454328    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.454329 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454331    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454335    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454338    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454342    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454345    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454347    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454351    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454354    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454357    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454360    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454362    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454370    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454374    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454376    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454379    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454381    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454385    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454388    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454391    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.454443 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454394    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454397    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454400    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454402    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454407    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454411    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454414    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454417    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454421    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454424    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454427    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454430    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454432    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454435    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454438    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454440    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454444    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454447    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454450    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.454931 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454453    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454456    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454459    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454462    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454464    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454467    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454470    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454473    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454477    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454480    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454482    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454485    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454487    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454490    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454494    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454497    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454500    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454503    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454505    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454508    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.455448 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454511    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454515    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454520    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454523    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454526    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454528    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454531    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454534    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454537    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454539    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454542    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454545    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454547    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454550    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454552    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454555    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454557    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454560    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454563    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454569    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.455941 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454572    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454575    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454578    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454581    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454583    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454586    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454589    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454592    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454594    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454598    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454600    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454603    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454606    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454608    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454611    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454613    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454616    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454618    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454621    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454623    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.456465 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454626    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454629    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454632    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454634    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454637    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.454639    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455423    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455432    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455437    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455440    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455443    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455446    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455449    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455453    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455456    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455459    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455462    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455464    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455467    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455470    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.456956 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455473    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455475    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455478    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455480    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455483    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455486    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455488    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455491    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455493    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455496    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455499    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455501    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455504    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455507    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455510    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455513    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455516    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455518    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455522    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455526    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.457504 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455529    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455533    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455536    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455539    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455543    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455546    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455549    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455551    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455554    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455556    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455559    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455561    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455564    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455566    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455569    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455571    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455574    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455576    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455579    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455581    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.458028 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455584    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455587    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455589    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455592    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455594    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455597    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455600    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455603    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455607    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455610    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455612    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455615    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455617    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455620    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455622    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455625    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455627    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455630    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455633    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.458516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455635    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455638    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455641    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455643    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455646    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455648    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455651    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455653    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455656    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455658    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455661    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455663    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455666    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455669    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455671    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455674    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455677    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455679    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455682    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455685    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.458988 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455687    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455690    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455693    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455697    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455700    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455703    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455706    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455709    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455712    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455715    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455717    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455720    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.455724    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456234    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456240    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456243    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456246    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456249    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456252    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.459494 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456255    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456258    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456262    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456265    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456268    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456271    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456274    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456277    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456280    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456283    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456286    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456288    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456291    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456294    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456296    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456299    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456302    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456305    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456307    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.459964 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456310    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456312    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456315    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456317    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456320    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456323    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456337    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456340    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456342    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456345    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456348    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456351    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456354    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456356    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456359    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456361    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456364    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456367    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456369    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.460522 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456372    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456375    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456377    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456380    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456383    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456385    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456388    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456391    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456393    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456405    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456408    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456410    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456413    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456416    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456419    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456422    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456424    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456427    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456433    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456437    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.461388 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456447    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456450    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456453    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456456    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456459    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456462    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456464    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456467    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456470    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456472    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456477    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456481    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456485    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456488    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456491    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456494    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456497    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456500    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456503    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456506    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.462291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456508    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456511    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456514    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456516    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456519    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456521    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456524    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456527    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456529    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456532    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456535    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456538    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456542    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456552    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456555    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456558    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456561    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456563    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456567    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456569    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.463204 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456572    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.456574    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456670    4942 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456677    4942 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456693    4942 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456698    4942 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456702    4942 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456705    4942 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456710    4942 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456715    4942 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456718    4942 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456721    4942 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456724    4942 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456727    4942 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456731    4942 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456734    4942 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456737    4942 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456740    4942 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456743    4942 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456746    4942 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456753    4942 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456756    4942 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456759    4942 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456763    4942 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:53.464044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456768    4942 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456772    4942 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456781    4942 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456784    4942 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456788    4942 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456791    4942 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456794    4942 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456797    4942 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456801    4942 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456804    4942 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456809    4942 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456812    4942 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456815    4942 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456818    4942 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456821    4942 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456824    4942 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456831    4942 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456834    4942 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456837    4942 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456840    4942 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456843    4942 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456847    4942 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456850    4942 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456853    4942 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456856    4942 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:53.465131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456859    4942 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456863    4942 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456866    4942 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456869    4942 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456872    4942 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456875    4942 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456878    4942 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456887    4942 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456891    4942 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456894    4942 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456903    4942 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456907    4942 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456910    4942 flags.go:64] FLAG: --help="false"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456913    4942 flags.go:64] FLAG: --hostname-override="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456916    4942 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456920    4942 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456923    4942 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456927    4942 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456930    4942 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456933    4942 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456936    4942 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456939    4942 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456942    4942 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:53.466327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456945    4942 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456948    4942 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456951    4942 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456954    4942 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456957    4942 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456960    4942 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456964    4942 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456967    4942 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456970    4942 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456973    4942 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456977    4942 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456983    4942 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456986    4942 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456989    4942 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456992    4942 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.456995    4942 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457000    4942 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457017    4942 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457022    4942 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457029    4942 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457040    4942 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457045    4942 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457048    4942 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457052    4942 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457055    4942 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:53.467505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457058    4942 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457061    4942 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457064    4942 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457067    4942 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457075    4942 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457078    4942 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457081    4942 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457084    4942 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457087    4942 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457090    4942 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457093    4942 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457096    4942 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457099    4942 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457102    4942 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02f0857ff1c50eacf"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457106    4942 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457109    4942 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457112    4942 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457117    4942 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457121    4942 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457124    4942 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457127    4942 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457130    4942 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457134    4942 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457138    4942 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457142    4942 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457145    4942 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:53.468587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457148    4942 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457150    4942 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457160    4942 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457163    4942 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457166    4942 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457169    4942 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457172    4942 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457175    4942 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457179    4942 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457182    4942 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457185    4942 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457188    4942 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457191    4942 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457194    4942 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457198    4942 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457200    4942 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457206    4942 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457209    4942 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457212    4942 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457243    4942 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457246    4942 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457250    4942 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457253    4942 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457258    4942 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457261    4942 flags.go:64] FLAG: --v="2"
Oct 30 18:36:53.469698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457265    4942 flags.go:64] FLAG: --version="false"
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457270    4942 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457274    4942 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457277    4942 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457471    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457475    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457478    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457481    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457484    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457487    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457496    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457502    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457505    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457508    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457511    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457514    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457517    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457519    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457522    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457525    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.470785 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457528    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457530    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457533    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457536    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457538    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457541    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457544    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457546    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457549    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457552    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457556    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457559    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457562    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457576    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457581    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457584    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457588    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457591    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457594    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457596    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.471535 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457599    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457602    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457605    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457614    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457617    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457619    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457622    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457625    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457628    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457631    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457633    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457636    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457639    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457641    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457644    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457647    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457649    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457652    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457654    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.472175 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457658    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457662    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457666    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457671    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457674    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457677    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457680    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457683    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457686    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457690    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457693    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457695    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457698    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457700    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457703    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457706    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457709    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457720    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457723    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.472881 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457726    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457729    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457732    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457734    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457737    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457740    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457751    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457755    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457758    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457761    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457764    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457766    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457769    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457772    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457774    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457777    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457781    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457784    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457786    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457789    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.473428 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457791    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457794    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457798    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457801    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457803    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457806    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457809    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457811    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457814    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457817    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457819    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457828    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.457854    4942 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457905    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457911    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457914    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.473903 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457918    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457921    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457924    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457927    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457929    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457932    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457935    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457937    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457940    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457943    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457945    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457948    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457951    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457955    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457958    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457961    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457963    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457966    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457970    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457973    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.474421 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457975    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457978    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457981    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457983    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457986    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457988    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457991    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.457994    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458002    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458020    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458023    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458025    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458028    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458031    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458034    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458036    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458039    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458042    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458045    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.475189 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458047    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458050    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458053    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458056    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458058    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458061    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458067    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458070    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458073    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458075    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458078    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458082    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458085    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458087    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458090    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458093    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458095    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458098    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458101    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458104    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.475697 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458107    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458109    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458118    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458121    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458124    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458126    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458129    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458132    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458136    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458140    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458143    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458146    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458149    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458152    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458154    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458157    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458160    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458162    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458168    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.476247 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458170    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458173    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458176    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458178    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458183    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458185    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458188    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458190    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458193    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458196    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458198    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458201    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458204    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458207    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458211    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458214    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458223    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458226    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458229    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458232    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.476720 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458235    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458237    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458241    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458243    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.458246    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.464426    4942 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.464453    4942 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.464482    4942 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.464491    4942 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464581    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464591    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464597    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464603    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464608    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464613    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464618    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464623    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464627    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464632    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464636    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.477250 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464641    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464645    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464649    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464654    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464659    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464663    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464668    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464673    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464678    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464682    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464689    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464695    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464700    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464705    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464710    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464714    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464719    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464728    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464733    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.477745 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464738    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464742    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464747    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464751    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464755    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464760    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464765    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464770    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464775    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464779    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464784    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464788    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464792    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464796    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464801    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464805    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464812    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464816    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464821    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464825    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.478576 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464829    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464834    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464838    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464843    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464847    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464852    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464858    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464864    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464869    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464874    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464879    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464883    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464887    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464892    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464896    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464901    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464906    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464910    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464915    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.479245 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464920    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464924    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464929    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464933    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464938    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464942    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464947    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464951    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464956    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464960    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464965    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464970    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464974    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464978    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464982    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464988    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464992    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.464997    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465001    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.479876 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465028    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465033    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465037    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465042    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465047    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465051    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465055    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465060    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465064    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465068    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465073    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465077    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465081    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465086    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465090    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465095    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465099    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.480429 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465103    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.465146    4942 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465389    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465397    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465403    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465408    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465413    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465419    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465423    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465428    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465435    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465441    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465448    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465452    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465457    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465461    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.480846 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465466    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465470    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465475    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465479    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465484    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465488    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465493    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465497    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465502    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465506    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465510    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465515    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465521    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465525    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465529    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465534    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465538    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465543    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465547    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465551    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.481255 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465555    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465560    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465564    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465568    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465574    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465579    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465583    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465588    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465592    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465596    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465600    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465607    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465613    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465618    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465623    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465628    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465632    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465636    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465641    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.481772 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465645    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465650    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465654    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465659    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465663    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465667    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465672    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465676    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465681    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465685    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465689    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465694    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465698    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465702    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465706    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465711    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465716    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465720    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465726    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.482272 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465730    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465735    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465740    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465744    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465748    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465752    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465757    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465762    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465766    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465771    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465775    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465780    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465784    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465788    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465793    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465797    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465801    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465806    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465810    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465814    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.483147 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465819    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465824    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465828    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465832    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465836    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465841    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465845    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465850    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465854    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465858    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465863    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465867    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465872    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465876    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.465915    4942 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465971    4942 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:53.483887 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465979    4942 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465983    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465988    4942 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465993    4942 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.465997    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466002    4942 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466023    4942 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466030    4942 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466040    4942 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466045    4942 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466049    4942 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466054    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466061    4942 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466066    4942 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466070    4942 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466075    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466079    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466084    4942 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466088    4942 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466093    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:53.484516 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466097    4942 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466101    4942 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466106    4942 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466110    4942 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466114    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466119    4942 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466123    4942 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466128    4942 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466132    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466137    4942 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466142    4942 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466147    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466151    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466155    4942 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466160    4942 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466164    4942 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466169    4942 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466173    4942 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466178    4942 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466183    4942 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:53.485153 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466189    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466194    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466198    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466203    4942 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466211    4942 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466217    4942 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466222    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466227    4942 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466232    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466236    4942 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466240    4942 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466245    4942 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466249    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466254    4942 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466259    4942 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466263    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466267    4942 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466272    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466278    4942 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:53.485640 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466282    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466286    4942 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466291    4942 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466296    4942 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466301    4942 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466305    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466309    4942 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466314    4942 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466318    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466323    4942 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466327    4942 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466331    4942 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466336    4942 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466342    4942 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466346    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466351    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466355    4942 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466361    4942 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466366    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466370    4942 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:53.486168 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466375    4942 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466379    4942 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466383    4942 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466388    4942 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466392    4942 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466396    4942 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466401    4942 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466405    4942 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466410    4942 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466414    4942 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466418    4942 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466423    4942 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466427    4942 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466434    4942 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466439    4942 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466443    4942 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466449    4942 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466453    4942 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466458    4942 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:53.486645 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466462    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466466    4942 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466471    4942 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466475    4942 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466480    4942 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466484    4942 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:53.466490    4942 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.466582    4942 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468171    4942 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468254    4942 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468535    4942 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468666    4942 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468716    4942 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:11 +0000 UTC" deadline="2028-07-19 14:26:44.571631768 +0000 UTC"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.468753    4942 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15067h49m51.102881043s"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.469298    4942 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.471025    4942 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:53.487128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.472367    4942 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:53.487560 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.474144    4942 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:53.487560 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.474814    4942 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:53.487560 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.476795    4942 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c083c8e-20a4-43e0-84f1-3614c4fe40f1:/dev/nvme0n1p3 8543e099-c652-4ddc-b640-72590b55d365:/dev/nvme0n1p4]
Oct 30 18:36:53.487674 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.476817    4942 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ba3faaf9f0c3bd94653b4af2cc0d4679a3704e78ccfba567d66edb6fa82daa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ba3faaf9f0c3bd94653b4af2cc0d4679a3704e78ccfba567d66edb6fa82daa/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f97130927c38ccb9e0355ffad92ec9e6ad9fd925751f47ee344b52d434e7f91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f97130927c38ccb9e0355ffad92ec9e6ad9fd925751f47ee344b52d434e7f91/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9e585e753d712a584ae7d5046878da28274cfe15bce41be2a25b0bcda951e39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9e585e753d712a584ae7d5046878da28274cfe15bce41be2a25b0bcda951e39/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adf53e3927bab969973feb3684fed133230756bd4fc55e5dab9d713080f08a17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adf53e3927bab969973feb3684fed133230756bd4fc55e5dab9d713080f08a17/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec4c24cbd3a6034c72a2429b90e556466240f90674a9214f3f4c5e175cf162bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec4c24cbd3a6034c72a2429b90e556466240f90674a9214f3f4c5e175cf162bd/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb9d479-2705-4c31-84c1-503e24ae3a0e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0eb9d479-2705-4c31-84c1-503e24ae3a0e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~projected/kube-api-access-4cntt:{mountpoint:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~projected/kube-api-access-4cntt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/200dacf8-0b68-43ea-87ec-e5f67becf695/volumes/kubernetes.io~projected/kube-api-access-cblvj:{mountpoint:/var/lib/kubelet/pods/200dacf8-0b68-43ea-87ec-e5f67becf695/volumes/kubernetes.io~projected/kube-api-access-cblvj major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27812221-bc2b-4915-ae7d-105a48913d66/volumes/kubernetes.io~projected/kube-api-access-vwm9z:{mountpoint:/var/lib/kubelet/pods/27812221-bc2b-4915-ae7d-105a48913d66/volumes/kubernetes.io~projected/kube-api-access-vwm9z major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28992bb6-3816-46b7-af81-3b1a3932432c/volumes/kubernetes.io~projected/kube-api-access-q75gw:{mountpoint:/var/lib/kubelet/pods/28992bb6-3816-46b7-af81-3b1a3932432c/volumes/kubernetes.io~projected/kube-api-access-q75gw major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb00238-6e40-41f7-b8c4-05f7ef60e913/volumes/kubernetes.io~projected/kube-api-access-7pkcl:{mountpoint:/var/lib/kubelet/pods/2cb00238-6e40-41f7-b8c4-05f7ef60e913/volumes/kubernetes.io~projected/kube-api-access-7pkcl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/kube-api-access-trnkl:{mountpoint:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/kube-api-access-trnkl major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~projected/kube-api-access-nv4nw:{mountpoint:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~projected/kube-api-access-nv4nw major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/kube-api-access-m8jxk:{mountpoint:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/kube-api-access-m8jxk major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44aaf253-a06d-4d3d-aac8-c1a6424dd260/volumes/kubernetes.io~projected/kube-api-access-r7f87:{mountpoint:/var/lib/kubelet/pods/44aaf253-a06d-4d3d-aac8-c1a6424dd260/volumes/kubernetes.io~projected/kube-api-access-r7f87 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c20dfb-2366-4678-8ffd-9ad49a7774fd/volumes/kubernetes.io~projected/kube-api-access-llbqz:{mountpoint:/var/lib/kubelet/pods/49c20dfb-2366-4678-8ffd-9ad49a7774fd/volumes/kubernetes.io~projected/kube-api-access-llbqz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~projected/kube-api-access-g4cwn:{mountpoint:/var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~projected/kube-api-access-g4cwn major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6914d9ae-4112-4a25-9f37-aa04aa12facc/volumes/kubernetes.io~projected/kube-api-access-6g28g:{mountpoint:/var/lib/kubelet/pods/6914d9ae-4112-4a25-9f37-aa04aa12facc/volumes/kubernetes.io~projected/kube-api-access-6g28g major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~projected/kube-api-access-tbpc5:{mountpoint:/var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~projected/kube-api-access-tbpc5 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~secret/serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c2d2e58-122c-47f9-8c32-0f73c4e8331c/volumes/kubernetes.io~projected/kube-api-access-djqnh:{mountpoint:/var/lib/kubelet/pods/8c2d2e58-122c-47f9-8c32-0f73c4e8331c/volumes/kubernetes.io~projected/kube-api-access-djqnh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~projected/kube-api-access-rh68w:{mountpoint:/var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~projected/kube-api-access-rh68w major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~projected/kube-api-access-wlsr4:{mountpoint:/var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~projected/kube-api-access-wlsr4 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a395f190-2d86-410f-81ef-93290e75b3a0/volumes/kubernetes.io~projected/kube-api-access-8skln:{mountpoint:/var/lib/kubelet/pods/a395f190-2d86-410f-81ef-93290e75b3a0/volumes/kubernetes.io~projected/kube-api-access-8skln major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~projected/kube-api-access-skd9s:{mountpoint:/var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~projected/kube-api-access-skd9s major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0c7ad43-3851-4baf-8b43-81597fa725b8/volumes/kubernetes.io~projected/kube-api-access-b624x:{mountpoint:/var/lib/kubelet/pods/b0c7ad43-3851-4baf-8b43-81597fa725b8/volumes/kubernetes.io~projected/kube-api-access-b624x major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904/volumes/kubernetes.io~projected/kube-api-access-9hnrh:{mountpoint:/var/lib/kubelet/pods/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904/volumes/kubernetes.io~projected/kube-api-access-9hnrh major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~projected/kube-api-access-ljdmf:{mountpoint:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~projected/kube-api-access-ljdmf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0c4450-cc6e-4124-9177-af902ae797ba/volumes/kubernetes.io~projected/kube-api-access-w7lpp:{mountpoint:/var/lib/kubelet/pods/cb0c4450-cc6e-4124-9177-af902ae797ba/volumes/kubernetes.io~projected/kube-api-access-w7lpp major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3132166-1ec1-448e-98c2-b9844cc8ef00/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d3132166-1ec1-448e-98c2-b9844cc8ef00/volumes/kubernetes.io~secret/agent-certs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e844bc94-4bbe-410b-9cb6-807b45c555df/volumes/kubernetes.io~projected/kube-api-access-fvh28:{mountpoint:/var/lib/kubelet/pods/e844bc94-4bbe-410b-9cb6-807b45c555df/volumes/kubernetes.io~projected/kube-api-access-fvh28 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8e07000-7744-4dc0-ab28-b877fe1d0cc8/volumes/kubernetes.io~projected/kube-api-access-9qg6m:{mountpoint:/var/lib/kubelet/pods/e8e07000-7744-4dc0-ab28-b877fe1d0cc8/volumes/kubernetes.io~projected/kube-api-access-9qg6m major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb/volumes/kubernetes.io~projected/kube-api-access-bcfxb:{mountpoint:/var/lib/kubelet/pods/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb/volumes/kubernetes.io~projected/kube-api-access-bcfxb major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb025a17-a0b0-4f54-9485-b8a85657a54f/volumes/kubernetes.io~projected/kube-api-access-559c2:{mountpoint:/var/lib/kubelet/pods/eb025a17-a0b0-4f54-9485-b8a85657a54f/volumes/kubernetes.io~projected/kube-api-access-559c2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f00a8cdd-93b4-459e-b271-03c501a4c5e5/volumes/kubernetes.io~projected/kube-api-access-s5p7s:{mountpoint:/var/lib/kubelet/pods/f00a8cdd-93b4-459e-b271-03c501a4c5e5/volumes/kubernetes.io~projected/kube-api-access-s5p7s major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5c18994-379c-4961-beac-4a01d127cef0/volumes/kubernetes.io~projected/kube-api-access-8t8nq:{mountpoint:/var/lib/kubelet/pods/f5c18994-379c-4961-beac-4a01d127cef0/volumes/kubernetes.io~projected/kube-api-access-8t8nq major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/192c03b71b6b691e1c43f806e269c0f0d742d8a9c38aeab7966d83ff5e57c323/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3e7b79ac4e15ff4a5f03f8b03b3f85355d8cfd98a5d71053b1ab351b0cb84c51/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ac15927a99f866a294b88d19add71ccf18c556f2894705436cfa10eb69d9de16/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b9688c9d28cf0a530891f08c48d0b13cee80d9f963bce7307ea2566e20161cfd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7f200623ef63517c6d50950c25f18909b53a34daf562fd7d108407d11e0ad41c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3cfa407290057125c059d1781cefa58896ea9b960197d0c40cdbe8e7c128e188/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2ed1116b1cf0dd3216ab1f14df526a52a8e4988e7db802cee3f9e704e0d42f75/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8dca963638f5c868cf2d9b59ed997ba4bbe408703601de1274156259c0f9e4c2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9052e6ddddeb22c0f3dc1db4dfe8685a11f27935f1648dbc2a6a517f7b5179d6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/704560d10e85e34450425dc69085ed92975f9511b0e2bd226faefe3a8df8d7e3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b5757b3a5303d650a6d26030c26244b480799fa9586887c1767e1eaf8b733058/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/1de4b367b00124083eb912ca63c2b8b7722c3d213c95668a2811f0e3952037ed/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bd23c95c0e485b475054d6ec6dc7d3c14d3f614340a91b8ddd0bf6aaee7528e7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/643a95d20c23649d5d6c1a191d16cae2411ce11dfae58476d1cb7e1b19b711f0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/12afb16ec39cf99cf4ad322e7b9e266aa41ffc87eefacda9aadd7386ae9e386b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/542c66a4ae3313071c9902c6525a4d876c5fe88a94e2bb36c743089f44a0e9e7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7bc9ba1fdf62811b96776801254eab027beb98e2bf877b7313c7a2697813c693/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4421c801286ccf28a4d610a7f6aff1afb2867729b6e8713408c8e0fd209c076f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4988b5ba0f977f6b7e70be9dd31d1ca75ea13ad8ed6872b9793b56d8b44ef7fd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f0a66d94fc6dea278efe219602d868af37214a58080b335c37133298fdddc1f2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a2bad3a3386dc25013c3fee75f9722204a7c9b9471350437198c0801b00ac922/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/dd900a00abb5514a8598efaab32794c99adb789da59a7df067ca618ee53e829c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/28c517a8a4c5ff6409feaabb19959c21665747aa192b0823efed265674da5bc8/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/3d42f02addefb43ec4911f837a82aa301e3785aab67dbf9d793a771f6cf81e24/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/93774a6fddf188dc237c4d25af2040ce3cf8118831ebe351e0003837235d5aef/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6a90d9bdd8e78ad4eb4bca85a4c1ef496c6d476b9f55abd7fa6f0291de3361a9/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/4d1ba1783e90a291d92489398fb644c5dd6d79ce8633d98a623c7f70eb11990c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2be20d69350e5a04d379f6e071508e7589cb167f2d4b5e1f796f678ba93a14c1/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/66f7cc6b6c9b7cf3db1bf2d4fa055dd65292b23a0a4bef38a8218d4568ac2439/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/46c1b93ef3194360d2b705f92d8e0eecb78b3f03521761e651daa1cde083058d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/09ae3a9c186d28958a90cc003e455d498194ea156fa53d35b3f1ddd7e2cf3732/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b5b215d7ec7b03dfbcc725f8dd416b2c8c31255b15f2f9d554d575f47281811b/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/47caa7cec93c266b03350283f5fddaf4a78fe30de6fd40f610df96818682079b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/44a8d7096acb3af405979e8c6e4cede998fd3cca30415378f4623f3092c8ccac/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/e2a2ab518bc3f1d84b936433688271a351a582129cc24166b29972aea306159e/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/72615df8f65cb78efdd035e082e4ab775762c490838171760e0491be0755f8a1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/9472a040900a4beef2cc8ea7f8a2e19c441ae9b172fe207f4999496e600bc283/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/90ab166597ad85dfa88f96508dd046ac5ecaa71ae9bff17799285745384f5d59/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/28a34dd1a5aafdfea3b53e148b61ee9d6c4e07ad0e053a5c4224bed5e208977a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/f7d01cba8b830352856d10cf9163da9182b4ac71decb9160b95b2e2ccc169801/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/698ba53b27ab14e712608b760f54927c80790253e3fd19b25944339954b3276c/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/ad5270bb1d0fcb2d93fc7181b9902660d40eebc75fd7fe9b8f4a2967ece140eb/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/368ee82fae1da4129a870805b0120b74c05bd804fc852b1db346c13a623d89e8/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/47fe1d659e800293b99214ecf50ef5706bac7e6cd8c49c1ee3047c6aa155cd57/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/31ac245f0ed894c5fd887ccd2620cc2294df2976fdb3f348a630ed106b7b5434/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/b5ee2aec43e66f15a2f0ff116e1fac1d4b291f30ca720ff36948bc2bbdadea74/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/6b405215bccea5f978a3c08bbc3617e45f074b8e16eb78f08405dba7711aab49/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9d5f73071be33d9a73fc5a988cab843e071481f52ba31607bf7d439caec60b39/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/aad0f5258cf353ef98bbdd59ba348b39b081ce2d81ef3029d2000c42a3452476/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0744e199dd056d3adb5a24dcb0eef82456ec262935be0ffbc5ebeb1f41fef2e3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/2a49dbab637ed429bb122111f8695ad065ba7628f8e1ae088275384f31700b39/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/3d60c11c59729ce80c51fe0fb14dbfee8e7f4b27e72c0aa53f47b703685e7d98/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/52ddc17409d4bd3b7957ab9f699000cdf6ebd1782b83d5264bb5331fa302144d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a5f95a021257cf8099af78c60455c5417da819762f63d203ef1f336afa51fa59/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e9618237c8e3e17d17ee8db3d5d643523859e2538993937313695cbbff22b310/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fa02d1304e2eb8dde2029e8923bd2a8f67733b0a65e74a990d7c1ef4f35e973c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.486542    4942 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:53.486068752 +0000 UTC m=+0.064164435 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110022 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d2000e31f1afacf01c497603e72c SystemUUID:ec27d200-0e31-f1af-acf0-1c497603e72c BootID:c8994aaa-a7aa-499f-9f00-230a502bcb7b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42813bc6877cf41bc6723c69fbe18cb03f80886a8af02f84652a7974b296baf8/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44aaf253-a06d-4d3d-aac8-c1a6424dd260/volumes/kubernetes.io~projected/kube-api-access-r7f87 DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f97130927c38ccb9e0355ffad92ec9e6ad9fd925751f47ee344b52d434e7f91/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9e585e753d712a584ae7d5046878da28274cfe15bce41be2a25b0bcda951e39/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0e8a87967858b0f35366a61ebba0e8615ae1a2533f8daf2b6112d5ed6180986/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~projected/kube-api-access-rh68w DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a10a2e1eb5430ef49f1a9e5d065f25f943117866d3e0d737e3b970ae89a0648/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6031797e3819ac8be270b41fef9930f9c95b89c1347ace52946048eeed92a5b1/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb025a17-a0b0-4f54-9485-b8a85657a54f/volumes/kubernetes.io~projected/kube-api-access-559c2 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec4c24cbd3a6034c72a2429b90e556466240f90674a9214f3f4c5e175cf162bd/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb00238-6e40-41f7-b8c4-05f7ef60e913/volumes/kubernetes.io~projected/kube-api-access-7pkcl DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3132166-1ec1-448e-98c2-b9844cc8ef00/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4db9f41e6aaae750832f47b1917d7d61a664a7099b94b3be8ba65accfb749839/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/kube-api-access-m8jxk DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/kube-api-access-trnkl DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a6f056f1c39065a4cc1de9ae9cd61c53d41e2d488b2e4650aac454eb53098f0/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c09baf1bf7d05ba92ae8fbf7496fcb9b605fd6b2d38cd456de5d7c5f7c71f9/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e844bc94-4bbe-410b-9cb6-807b45c555df/volumes/kubernetes.io~projected/kube-api-access-fvh28 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ba3faaf9f0c3bd94653b4af2cc0d4679a3704e78ccfba567d66edb6fa82daa/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb9d479-2705-4c31-84c1-503e24ae3a0e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/volumes/kubernetes.io~projected/kube-api-access-g4cwn DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb/volumes/kubernetes.io~projected/kube-api-access-bcfxb DeviceMajor:0 DeviceMinor:323 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6914d9ae-4112-4a25-9f37-aa04aa12facc/volumes/kubernetes.io~projected/kube-api-access-6g28g DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a395f190-2d86-410f-81ef-93290e75b3a0/volumes/kubernetes.io~projected/kube-api-access-8skln DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00cbe9a808bf6d23066f07b20f0ec6a0d74fc4c8798b2e7a8ccf467aa8c4234b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adf53e3927bab969973feb3684fed133230756bd4fc55e5dab9d713080f08a17/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed6b9889345a17e86c438930ffa951c6af187eddc25af9677ff00d9de4b4eae0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8156c31eb6f83c0b184d0c9eed70d331075736dcea75ffbd816b652075893c08/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/200dacf8-0b68-43ea-87ec-e5f67becf695/volumes/kubernetes.io~projected/kube-api-access-cblvj DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5c18994-379c-4961-beac-4a01d127cef0/volumes/kubernetes.io~projected/kube-api-access-8t8nq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f00a8cdd-93b4-459e-b271-03c501a4c5e5/volumes/kubernetes.io~projected/kube-api-access-s5p7s DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c2d2e58-122c-47f9-8c32-0f73c4e8331c/volumes/kubernetes.io~projected/kube-api-access-djqnh DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e93dbed026ab91677cfd434b0c68e6db5f685f455982b5cc1d4b9af29f95d495/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/748f89ab39ec8924c8bb671790d9039fdc06ad8d5790436804fca0acf0f7cf41/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volumes/kubernetes.io~projected/kube-api-access-4cntt DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9506e6a-0145-4e49-b3f1-acd79fc488f5/volumes/kubernetes.io~projected/kube-api-access-skd9s DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/363e48de-1ba5-4934-a2e7-e38cae066bb8/volumes/kubernetes.io~projected/kube-api-access-nv4nw DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/708f16f0-f04f-4ea8-8e51-a1d2ca19858a/volumes/kubernetes.io~projected/kube-api-access-tbpc5 DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a09bbce7-7e03-435e-992a-c1d89acdbe5a/volumes/kubernetes.io~projected/kube-api-access-wlsr4 DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904/volumes/kubernetes.io~projected/kube-api-access-9hnrh DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0c4450-cc6e-4124-9177-af902ae797ba/volumes/kubernetes.io~projected/kube-api-access-w7lpp DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/49c20dfb-2366-4678-8ffd-9ad49a7774fd/volumes/kubernetes.io~projected/kube-api-access-llbqz DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec63d7a5b5e9fddb9a72dd7116ebb5cd417251e524a2f6f8e548a6f7850d85f2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57962e740fb832b85fd214da8c6eab49e5e15b6f506e734dedc29929285993b5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/111c9d09-fd7b-4a3f-99a7-3075467e5e58/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8e07000-7744-4dc0-ab28-b877fe1d0cc8/volumes/kubernetes.io~projected/kube-api-access-9qg6m DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0c7ad43-3851-4baf-8b43-81597fa725b8/volumes/kubernetes.io~projected/kube-api-access-b624x DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f806a99d39e0d61252459c5bf3476023cbf739f2f20aec1d1b96fe354cfc8460/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df11f17735b042bccd4c3b9c4563f245db5c9d89eeeec0e6039f37e54546c263/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27812221-bc2b-4915-ae7d-105a48913d66/volumes/kubernetes.io~projected/kube-api-access-vwm9z DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28992bb6-3816-46b7-af81-3b1a3932432c/volumes/kubernetes.io~projected/kube-api-access-q75gw DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6/volumes/kubernetes.io~projected/kube-api-access-ljdmf DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:42813bc6877cf41 MacAddress:7a:a0:a2:80:42:47 Speed:10000 Mtu:8901} {Name:4a10a2e1eb5430e MacAddress:0a:67:76:dd:3e:5e Speed:10000 Mtu:8901} {Name:6031797e3819ac8 MacAddress:3a:4a:3c:79:c5:27 Speed:10000 Mtu:8901} {Name:62ba3faaf9f0c3b MacAddress:d6:40:5c:c4:c6:b3 Speed:10000 Mtu:8901} {Name:748f89ab39ec892 MacAddress:0e:e6:20:60:46:03 Speed:10000 Mtu:8901} {Name:7f97130927c38cc MacAddress:02:cf:46:c3:29:2a Speed:10000 Mtu:8901} {Name:8156c31eb6f83c0 MacAddress:72:4a:fb:df:39:d6 Speed:10000 Mtu:8901} {Name:8a6f056f1c39065 MacAddress:d2:6c:36:dc:da:e5 Speed:10000 Mtu:8901} {Name:a9e585e753d712a MacAddress:c2:8a:a7:ec:90:89 Speed:10000 Mtu:8901} {Name:adf53e3927bab96 MacAddress:e6:5a:bd:5e:b4:20 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f8:21:e4:cb Speed:0 Mtu:9001} {Name:br-int MacAddress:22:d8:4a:5f:e7:9a Speed:0 Mtu:8901} {Name:d0e8a87967858b0 MacAddress:7a:77:42:b5:5c:f5 Speed:10000 Mtu:8901} {Name:e93dbed026ab916 MacAddress:66:68:71:85:40:6d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f8:21:e4:cb Speed:0 Mtu:9001} {Name:f806a99d39e0d61 MacAddress:76:a8:20:25:f9:f6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0e:53:28:5d:04:b1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:78:d4:4d:ab:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.486908    4942 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.486958    4942 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487282    4942 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487308    4942 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-34.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487448    4942 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487456    4942 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487466    4942 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487481    4942 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487502    4942 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487675    4942 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487792    4942 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487831    4942 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487844    4942 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487855    4942 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487863    4942 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:53.487887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.487872    4942 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:53.488693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.488580    4942 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:53.488693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.488590    4942 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:53.489152 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.489130    4942 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:53.491319 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491304    4942 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:53.491319 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491318    4942 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:53.491433 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491423    4942 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:36:53.491622 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491612    4942 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:53.491653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491641    4942 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:53.491740 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491730    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491744    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491751    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491757    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491763    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491769    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:53.491776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491774    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491779    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491787    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491793    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491801    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491810    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491833    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:53.491957 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.491839    4942 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:53.495366 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.495351    4942 server.go:1263] "Started kubelet"
Oct 30 18:36:53.495598 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.495458    4942 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:53.495598 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.495488    4942 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:53.495598 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.495559    4942 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:53.495913 ip-10-0-134-34 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:53.496108 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.496082    4942 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:53.497655 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.497407    4942 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:53.502555 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.502522    4942 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:53.503040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.502746    4942 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:53.503040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.502769    4942 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:53.503040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.502945    4942 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:53.503040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.502752    4942 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:53.505040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.503590    4942 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:13 +0000 UTC" deadline="2028-08-09 08:30:22.894116412 +0000 UTC"
Oct 30 18:36:53.505040 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.503635    4942 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15565h53m29.390485453s"
Oct 30 18:36:53.505040 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.504210    4942 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-34.us-east-2.compute.internal\" not found"
Oct 30 18:36:53.507119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507034    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:53.507119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507075    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363e48de-1ba5-4934-a2e7-e38cae066bb8" volumeName="kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate" seLinuxMountContext=""
Oct 30 18:36:53.507119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507089    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c2d2e58-122c-47f9-8c32-0f73c4e8331c" volumeName="kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh" seLinuxMountContext=""
Oct 30 18:36:53.507119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507101    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8e07000-7744-4dc0-ab28-b877fe1d0cc8" volumeName="kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m" seLinuxMountContext=""
Oct 30 18:36:53.507119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507113    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363e48de-1ba5-4934-a2e7-e38cae066bb8" volumeName="kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507134    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="606559f4-9044-4445-90ee-d29a202d2b43" volumeName="kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507145    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111c9d09-fd7b-4a3f-99a7-3075467e5e58" volumeName="kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507157    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111c9d09-fd7b-4a3f-99a7-3075467e5e58" volumeName="kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507171    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507182    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44aaf253-a06d-4d3d-aac8-c1a6424dd260" volumeName="kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507198    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0c4450-cc6e-4124-9177-af902ae797ba" volumeName="kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507210    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f00a8cdd-93b4-459e-b271-03c501a4c5e5" volumeName="kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507244    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507258    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" volumeName="kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507281    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6914d9ae-4112-4a25-9f37-aa04aa12facc" volumeName="kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507293    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507310    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3132166-1ec1-448e-98c2-b9844cc8ef00" volumeName="kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507324    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb9d479-2705-4c31-84c1-503e24ae3a0e" volumeName="kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret" seLinuxMountContext=""
Oct 30 18:36:53.507338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507335    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c20dfb-2366-4678-8ffd-9ad49a7774fd" volumeName="kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507348    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9506e6a-0145-4e49-b3f1-acd79fc488f5" volumeName="kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507359    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb025a17-a0b0-4f54-9485-b8a85657a54f" volumeName="kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507371    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507384    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44aaf253-a06d-4d3d-aac8-c1a6424dd260" volumeName="kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507397    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9506e6a-0145-4e49-b3f1-acd79fc488f5" volumeName="kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507409    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5c18994-379c-4961-beac-4a01d127cef0" volumeName="kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507420    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28992bb6-3816-46b7-af81-3b1a3932432c" volumeName="kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507433    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb025a17-a0b0-4f54-9485-b8a85657a54f" volumeName="kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507444    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27812221-bc2b-4915-ae7d-105a48913d66" volumeName="kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507470    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507481    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111c9d09-fd7b-4a3f-99a7-3075467e5e58" volumeName="kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507501    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111c9d09-fd7b-4a3f-99a7-3075467e5e58" volumeName="kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507513    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200dacf8-0b68-43ea-87ec-e5f67becf695" volumeName="kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507536    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c20dfb-2366-4678-8ffd-9ad49a7774fd" volumeName="kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507548    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6914d9ae-4112-4a25-9f37-aa04aa12facc" volumeName="kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507561    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="708f16f0-f04f-4ea8-8e51-a1d2ca19858a" volumeName="kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507572    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507583    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507595    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507606    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507617    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="708f16f0-f04f-4ea8-8e51-a1d2ca19858a" volumeName="kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507628    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507640    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9506e6a-0145-4e49-b3f1-acd79fc488f5" volumeName="kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507651    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0c7ad43-3851-4baf-8b43-81597fa725b8" volumeName="kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507663    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbfe361-0dd8-4e1e-bf8f-cd48924ec904" volumeName="kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507676    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" volumeName="kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507688    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" volumeName="kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507700    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" volumeName="kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507712    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09bbce7-7e03-435e-992a-c1d89acdbe5a" volumeName="kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507725    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e844bc94-4bbe-410b-9cb6-807b45c555df" volumeName="kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28" seLinuxMountContext=""
Oct 30 18:36:53.507727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507738    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb" volumeName="kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507750    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="708f16f0-f04f-4ea8-8e51-a1d2ca19858a" volumeName="kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507762    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbfe361-0dd8-4e1e-bf8f-cd48924ec904" volumeName="kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507775    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf92a54-eb83-4b14-a48c-6a1fea2e11d6" volumeName="kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507786    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3132166-1ec1-448e-98c2-b9844cc8ef00" volumeName="kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507800    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8e07000-7744-4dc0-ab28-b877fe1d0cc8" volumeName="kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507811    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb025a17-a0b0-4f54-9485-b8a85657a54f" volumeName="kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507832    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f00a8cdd-93b4-459e-b271-03c501a4c5e5" volumeName="kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507844    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507856    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27812221-bc2b-4915-ae7d-105a48913d66" volumeName="kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507868    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507879    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c20dfb-2366-4678-8ffd-9ad49a7774fd" volumeName="kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507915    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09bbce7-7e03-435e-992a-c1d89acdbe5a" volumeName="kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507931    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbfe361-0dd8-4e1e-bf8f-cd48924ec904" volumeName="kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507944    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf92a54-eb83-4b14-a48c-6a1fea2e11d6" volumeName="kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507956    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb00238-6e40-41f7-b8c4-05f7ef60e913" volumeName="kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507968    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507980    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c20dfb-2366-4678-8ffd-9ad49a7774fd" volumeName="kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.507992    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a395f190-2d86-410f-81ef-93290e75b3a0" volumeName="kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508019    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111c9d09-fd7b-4a3f-99a7-3075467e5e58" volumeName="kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508034    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363e48de-1ba5-4934-a2e7-e38cae066bb8" volumeName="kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508047    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508058    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508070    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf92a54-eb83-4b14-a48c-6a1fea2e11d6" volumeName="kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508082    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508092    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="327823ad-e078-443a-a05d-70f1bdb181e8" volumeName="kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508105    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" volumeName="kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508120    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a395f190-2d86-410f-81ef-93290e75b3a0" volumeName="kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508132    4942 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" volumeName="kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508144    4942 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508151    4942 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508250    4942 factory.go:153] Registering CRI-O factory
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508267    4942 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508319    4942 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508326    4942 factory.go:55] Registering systemd factory
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508331    4942 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508351    4942 factory.go:103] Registering Raw factory
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508360    4942 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:53.508927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.508640    4942 manager.go:319] Starting recovery of all containers
Oct 30 18:36:53.513516 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.513490    4942 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:53.514783 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.514757    4942 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:53.514916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.514873    4942 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:53.516064 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.516043    4942 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:36:53.529337 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.529310    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:53.529456 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.529350    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:53.534083 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.534045    4942 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:53.535739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.535714    4942 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:53.535739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.535736    4942 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:53.535912 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.535758    4942 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:53.535912 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.535799    4942 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:36:53.543615 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.543582    4942 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:53.558086 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558062    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="bd4d3ed5d9aa0e4d5a435abe1f17b3f6611d84cca74f2a8aa0c95711985cf9fa" exitCode=0
Oct 30 18:36:53.558086 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558083    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="10e0c2776fbb63710f49853208d5e7f3df6c58448cd28bf301f783acbb413464" exitCode=0
Oct 30 18:36:53.558086 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558090    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="fd7c9661600bda420ff1e3fd71b34bed17d90436b4ee06823ef141445a20ba37" exitCode=0
Oct 30 18:36:53.558086 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558096    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="f8337f71bce69a765455503f007fd60ee0f7b2f58c71a99c8ceecade18b72dc3" exitCode=0
Oct 30 18:36:53.558314 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558104    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="1a999e91bc4aac8d69b22bcf4179ac75b3879194c2db26e9ed08693e70d8e70e" exitCode=0
Oct 30 18:36:53.558314 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.558112    4942 generic.go:356] "Generic (PLEG): container finished" podID="49c20dfb-2366-4678-8ffd-9ad49a7774fd" containerID="762cfeead1d81e98292ab1b76dbe998f1f34ca357870e7b54d1d99346d34c1e7" exitCode=0
Oct 30 18:36:53.589476 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.589445    4942 generic.go:356] "Generic (PLEG): container finished" podID="c5d63ad00e5a9245d4df7ea9f3aeb927" containerID="95bd95d3c619ce02445d7e62ef41bacb88cf97e80697aa5d204741e37229ec06" exitCode=0
Oct 30 18:36:53.593135 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.593115    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:53.593249 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.593139    4942 generic.go:356] "Generic (PLEG): container finished" podID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d" exitCode=255
Oct 30 18:36:53.600778 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.600730    4942 manager.go:324] Recovery completed
Oct 30 18:36:53.621525 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621503    4942 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:53.621525 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621521    4942 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:53.621686 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621539    4942 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:53.621686 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621668    4942 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621679    4942 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621698    4942 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621703    4942 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621709    4942 policy_none.go:50] "Start"
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621721    4942 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:53.621744 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621730    4942 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:53.621904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621852    4942 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:36:53.621904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621859    4942 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:36:53.621904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.621865    4942 policy_none.go:44] "Start"
Oct 30 18:36:53.626278 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.626253    4942 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:53.628955 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.628942    4942 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:53.629557 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.628967    4942 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:53.629631 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.629560    4942 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:53.629833 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.629821    4942 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:53.629881 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.629834    4942 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:53.629942 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.629931    4942 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:53.630168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.630052    4942 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:53.630168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.630120    4942 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:53.630168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.630125    4942 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:53.630330 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.630167    4942 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:53.630085113Z","UUID":"aae23d87-9c23-44e9-afe6-4f96544f07b0","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:53.631425 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.631408    4942 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:53.631511 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.631468    4942 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:53.631511 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.631475    4942 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:53.635868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.635836    4942 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"]
Oct 30 18:36:53.636493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636460    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"76487939913b7836b4b3b35bf9597886","Type":"ContainerStarted","Data":"b1f292a956c45e694561402bbfb1e575f7917c17150540fe06453bd052b2e4b4"}
Oct 30 18:36:53.636588 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636505    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"76487939913b7836b4b3b35bf9597886","Type":"ContainerStarted","Data":"42be654de88fe6ad99eee12250590b65a699dd8c1f041f87188983adad7483fd"}
Oct 30 18:36:53.636588 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636553    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerStarted","Data":"8c0e470012920b7f7435d9d1a4758ff59f69d4a9f46fa815a4b88e1f1b89595c"}
Oct 30 18:36:53.636588 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636573    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerDied","Data":"95bd95d3c619ce02445d7e62ef41bacb88cf97e80697aa5d204741e37229ec06"}
Oct 30 18:36:53.636714 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636592    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal" event={"ID":"c5d63ad00e5a9245d4df7ea9f3aeb927","Type":"ContainerStarted","Data":"a196db9dc6e0f3a5669ebeb957912e3c8f08d030442f1c9e6bc9fa3ca6a2aaa6"}
Oct 30 18:36:53.636714 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636709    4942 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.636786 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.636719    4942 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.651151 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.651126    4942 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:53.651247 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.651189    4942 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.651896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.651881    4942 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:53.651944 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:53.651923    4942 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.717347 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.717312    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.717497 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.717352    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.717497 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.717370    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.730440 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.730412    4942 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:53.731714 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.731698    4942 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:53.731801 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.731731    4942 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:53.731801 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.731748    4942 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-34.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:53.731886 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.731806    4942 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.745577 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.745551    4942 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.745667 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.745632    4942 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817662 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817631    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817684    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817706    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817641    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817746    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76487939913b7836b4b3b35bf9597886-config\") pod \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"76487939913b7836b4b3b35bf9597886\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:53.817901 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:53.817780    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d63ad00e5a9245d4df7ea9f3aeb927-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" (UID: \"c5d63ad00e5a9245d4df7ea9f3aeb927\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:54.488581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.488552    4942 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:54.505407 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.505384    4942 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:54.506259 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.506226    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc","openshift-image-registry/node-ca-96jgv","kube-system/global-pull-secret-syncer-qw889","openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw","kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-sfglz","openshift-multus/multus-additional-cni-plugins-bxfx7","openshift-network-diagnostics/network-check-target-lz7bk","openshift-ovn-kubernetes/ovnkube-node-njg9l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc","openshift-insights/insights-operator-7c4986d449-fs9gh","openshift-console-operator/console-operator-7cd56b8dbc-spj7q","openshift-ingress/router-default-77bc686d65-bdnkc","openshift-network-console/networking-console-plugin-86c478bb88-8pmcl","openshift-dns/dns-default-rc44q","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv","kube-system/kas-connection-checker-685895c7c9-j9q9f","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d","kube-system/kas-connection-checker-685895c7c9-64w7x","openshift-ingress-canary/ingress-canary-5cdsk","openshift-cluster-node-tuning-operator/tuned-8z7tf","openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn","kube-system/kas-connection-checker-685895c7c9-7v9n7","kube-system/konnectivity-agent-x5hb6","openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb","openshift-multus/multus-94k6t","openshift-network-operator/iptables-alerter-wqk77","openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557","openshift-image-registry/image-registry-574c7785-ktt9b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal","openshift-dns/node-resolver-flzfd","openshift-insights/insights-runtime-extractor-4x55m"]
Oct 30 18:36:54.510373 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.510285    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.510479 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.510423    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.511335 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.511315    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.511938 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.511914    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:54.512068 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512053    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qfk8m\""
Oct 30 18:36:54.512164 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512146    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:54.512233 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512162    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.512233 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512186    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.512317 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512246    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.512317 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.512265    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.513388 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.513369    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.514100 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.514082    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:54.517480 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517459    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b8djd\""
Oct 30 18:36:54.517556 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517490    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.517556 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517496    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:54.517556 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517509    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:54.517556 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517537    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.517696 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517591    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.517696 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517613    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:54.517750 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.517703    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:54.518176 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518157    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:54.518265 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518191    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.518488 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518473    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:54.518655 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518637    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.518842 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518822    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:54.518906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518848    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:54.518906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518859    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.518906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.518889    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wskq9\""
Oct 30 18:36:54.519053 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.519034    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:54.519183 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.519168    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:54.520973 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.520954    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.522090 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.522074    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.523220 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.523198    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.525091 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.525064    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:54.525219 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.525199    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:54.525292 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.525269    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.525369 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.525350    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.525419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.525374    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.528347 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528326    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:54.528605 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528584    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:54.528605 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528602    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:54.528698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528680    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:54.528698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528688    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:54.528765 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528698    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:54.528765 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.528755    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:54.529995 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.529978    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:54.530311 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.530282    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:54.530374 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.530320    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdfmc\""
Oct 30 18:36:54.530374 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.530345    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nzpfq\""
Oct 30 18:36:54.532338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532286    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-829tl\""
Oct 30 18:36:54.532338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532294    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lfspj\""
Oct 30 18:36:54.532338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532309    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4k7wf\""
Oct 30 18:36:54.532338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532300    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rwngh\""
Oct 30 18:36:54.532338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532339    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-psdnm\""
Oct 30 18:36:54.532513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532361    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-whw2z\""
Oct 30 18:36:54.532513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532386    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mkp5v\""
Oct 30 18:36:54.532513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532406    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jfgwk\""
Oct 30 18:36:54.532513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532432    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-82qzs\""
Oct 30 18:36:54.532513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532455    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqtqh\""
Oct 30 18:36:54.532648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532528    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:54.532648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532545    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:54.532648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532528    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:54.532648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532562    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9gnp\""
Oct 30 18:36:54.532835 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532812    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5twcv\""
Oct 30 18:36:54.532913 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532879    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2njn\""
Oct 30 18:36:54.532974 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.532937    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:54.533045 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533027    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-khvcj\""
Oct 30 18:36:54.533102 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533072    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v8cvv\""
Oct 30 18:36:54.533149 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533117    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:54.533199 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533188    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bskc2\""
Oct 30 18:36:54.533418 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533404    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:54.533449 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533420    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.533800 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533787    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mrncz\""
Oct 30 18:36:54.533868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533823    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x9qkh\""
Oct 30 18:36:54.533868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533826    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.533868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533836    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-75g8c\""
Oct 30 18:36:54.533868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533866    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533888    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533908    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533967    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533974    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxh65\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.533977    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534003    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534056    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534089    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534140    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534155    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534156    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534147    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.534333 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.534180    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.537816 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.537797    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:54.537816 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.537814    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r4rtj\""
Oct 30 18:36:54.597257 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.597231    4942 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:54.597417 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.597359    4942 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:54.605214 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.605186    4942 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:54.609809 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.609793    4942 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:54.609864 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.609837    4942 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:54.610675 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.610660    4942 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:54.610733 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.610693    4942 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:36:54.612442 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.612427    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:54.612907 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.612890    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.612967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.612895    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.612967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.612924    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:54.613739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613720    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:54.613841 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613823    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.613841 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613837    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613768    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613793    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613863    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613803    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-82wlx\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613885    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613895    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613835    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613910    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613856    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613800    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613867    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.613940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613868    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613877    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613782    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613814    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613720    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613842    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.613990    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:54.614287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.614262    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:54.621510 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621490    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.621585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621518    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.621585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621535    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.621585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621550    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:54.621585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621566    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.621585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621581    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.621776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621708    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.621820 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621781    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:54.621820 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621813    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.621918 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621856    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.621967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621925    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-cni-binary-copy\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.622031 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.621961    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.622090 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622028    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3132166-1ec1-448e-98c2-b9844cc8ef00-agent-certs\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:54.622090 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622072    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-default-certificate\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.622090 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622078    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.622231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622128    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.622231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622152    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-service-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.622231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622158    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.622231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622198    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.622231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622219    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.622432 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622236    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.622432 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622213    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-script-lib\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.622432 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622288    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.622432 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622357    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75gw\" (UniqueName: \"kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw\") pod \"kas-connection-checker-685895c7c9-j9q9f\" (UID: \"28992bb6-3816-46b7-af81-3b1a3932432c\") " pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:54.622558 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622517    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.622558 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622551    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.622648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622576    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.622648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622623    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.622726 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622655    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.622726 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622710    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.622780 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622733    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-serving-cert\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.622780 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622748    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cntt\" (UniqueName: \"kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.622837 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622780    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.622837 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622798    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.622894 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622848    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.622923 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.622895    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.623052 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623038    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.623052 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623038    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.623117 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623077    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.623117 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623109    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.623174 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623133    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.623174 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623168    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.623232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623197    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:54.623232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623223    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpc5\" (UniqueName: \"kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.623295 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623286    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.623325 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623302    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lpp\" (UniqueName: \"kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp\") pod \"network-check-source-6f78d59f99-8tvw8\" (UID: \"cb0c4450-cc6e-4124-9177-af902ae797ba\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:54.623361 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623318    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:54.623361 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623356    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qg6m\" (UniqueName: \"kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.623433 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623405    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkcl\" (UniqueName: \"kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl\") pod \"kas-connection-checker-685895c7c9-7v9n7\" (UID: \"2cb00238-6e40-41f7-b8c4-05f7ef60e913\") " pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:54.623468 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623458    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09bbce7-7e03-435e-992a-c1d89acdbe5a-cert\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:54.623512 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623490    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.623609 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.623659 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623627    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3132166-1ec1-448e-98c2-b9844cc8ef00-konnectivity-ca\") pod \"konnectivity-agent-x5hb6\" (UID: \"d3132166-1ec1-448e-98c2-b9844cc8ef00\") " pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:36:54.623659 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623635    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skd9s\" (UniqueName: \"kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.623724 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623671    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.623803 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623785    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.623850 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623834    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.623902 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623863    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-tmp\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.623902 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623885    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.623998 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623962    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.623998 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.623972    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-tmp-dir\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.624099 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624003    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.624099 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624061    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.624099 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624077    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-serving-cert\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.624099 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624089    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624111    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624128    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624165    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624194    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624223    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624256    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.624266 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624261    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4nw\" (UniqueName: \"kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624286    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-env-overrides\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624307    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlsr4\" (UniqueName: \"kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624315    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624408    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624481    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f00a8cdd-93b4-459e-b271-03c501a4c5e5-serviceca\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624523    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.624580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624572    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.624932 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624915    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.624989 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.624963    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.625051 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625025    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.625097 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625064    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.625140 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625095    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.625140 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625111    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-serving-cert\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.625140 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625122    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.625277 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625177    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.625277 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625244    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44aaf253-a06d-4d3d-aac8-c1a6424dd260-data-volume\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.625277 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625258    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.625397 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625321    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbqz\" (UniqueName: \"kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.625454 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625436    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-config-volume\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.625499 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625450    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.625544 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625496    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.625544 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625499    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.625850 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625829    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-snapshots\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.625931 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625891    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559c2\" (UniqueName: \"kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.625931 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625921    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.626062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.625977    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.626062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626035    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.626191 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626063    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.626191 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626098    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblvj\" (UniqueName: \"kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj\") pod \"volume-data-source-validator-58c9f5f46d-qc5dv\" (UID: \"200dacf8-0b68-43ea-87ec-e5f67becf695\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:54.626298 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626191    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.626298 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626223    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:54.626298 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626257    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.626298 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626289    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh68w\" (UniqueName: \"kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626322    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626354    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnrh\" (UniqueName: \"kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626384    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626412    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626528    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.626581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626563    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626625    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7f87\" (UniqueName: \"kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626660    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626691    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626726    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cwn\" (UniqueName: \"kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626757    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626788    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626813    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626845    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626876    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.626905 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626907    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626935    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdmf\" (UniqueName: \"kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626964    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.626994    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627212    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627244    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627276    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627313    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627345    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627376    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627403    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627435    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627467    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.627504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627500    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627534    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627591    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8nq\" (UniqueName: \"kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627624    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627662    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627694    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627720    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g28g\" (UniqueName: \"kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627752    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwm9z\" (UniqueName: \"kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627783    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627816    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skln\" (UniqueName: \"kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627848    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627875    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627908    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627940    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.627984    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628026    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628060    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628092    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfxb\" (UniqueName: \"kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb\") pod \"migrator-66df5d496b-8x557\" (UID: \"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628124    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628155    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5p7s\" (UniqueName: \"kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628180    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628211    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.628457 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628245    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628742    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-tmp\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628745    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628771    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.628933    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9506e6a-0145-4e49-b3f1-acd79fc488f5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629024    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-daemon-config\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629170    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-config\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.629503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629421    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovn-node-metrics-cert\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.629809 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629538    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-stats-auth\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.629885 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629836    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/27812221-bc2b-4915-ae7d-105a48913d66-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.630057 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630030    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.630057 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.629310    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-trusted-ca\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.630337 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630318    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/606559f4-9044-4445-90ee-d29a202d2b43-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:54.630584 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630565    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6914d9ae-4112-4a25-9f37-aa04aa12facc-tmp-dir\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.630696 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630672    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.630755 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630739    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9506e6a-0145-4e49-b3f1-acd79fc488f5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.630811 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630762    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvh28\" (UniqueName: \"kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28\") pod \"kas-connection-checker-685895c7c9-64w7x\" (UID: \"e844bc94-4bbe-410b-9cb6-807b45c555df\") " pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:54.630811 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630772    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.630811 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630806    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.630958 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630878    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.630958 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.630944    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.631081 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631055    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-config\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.631132 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631085    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-tuned\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.631235 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631109    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.631235 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631200    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.631235 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631224    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a395f190-2d86-410f-81ef-93290e75b3a0-iptables-alerter-script\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.631362 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631242    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.631362 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631273    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.631681 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631661    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.631756 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631733    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.631851 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631824    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0eb9d479-2705-4c31-84c1-503e24ae3a0e-original-pull-secret\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.631851 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631824    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.631939 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631917    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/111c9d09-fd7b-4a3f-99a7-3075467e5e58-ovnkube-config\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.631939 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631933    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqnh\" (UniqueName: \"kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.632033 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.631995    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.636473 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.636450    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75gw\" (UniqueName: \"kubernetes.io/projected/28992bb6-3816-46b7-af81-3b1a3932432c-kube-api-access-q75gw\") pod \"kas-connection-checker-685895c7c9-j9q9f\" (UID: \"28992bb6-3816-46b7-af81-3b1a3932432c\") " pod="kube-system/kas-connection-checker-685895c7c9-j9q9f"
Oct 30 18:36:54.637276 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.637256    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.645102 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.645083    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkcl\" (UniqueName: \"kubernetes.io/projected/2cb00238-6e40-41f7-b8c4-05f7ef60e913-kube-api-access-7pkcl\") pod \"kas-connection-checker-685895c7c9-7v9n7\" (UID: \"2cb00238-6e40-41f7-b8c4-05f7ef60e913\") " pod="kube-system/kas-connection-checker-685895c7c9-7v9n7"
Oct 30 18:36:54.650779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.650762    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b624x\" (UniqueName: \"kubernetes.io/projected/b0c7ad43-3851-4baf-8b43-81597fa725b8-kube-api-access-b624x\") pod \"network-check-target-lz7bk\" (UID: \"b0c7ad43-3851-4baf-8b43-81597fa725b8\") " pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:54.651469 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.651446    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlsr4\" (UniqueName: \"kubernetes.io/projected/a09bbce7-7e03-435e-992a-c1d89acdbe5a-kube-api-access-wlsr4\") pod \"ingress-canary-5cdsk\" (UID: \"a09bbce7-7e03-435e-992a-c1d89acdbe5a\") " pod="openshift-ingress-canary/ingress-canary-5cdsk"
Oct 30 18:36:54.651545 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.651513    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skd9s\" (UniqueName: \"kubernetes.io/projected/a9506e6a-0145-4e49-b3f1-acd79fc488f5-kube-api-access-skd9s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-79v5d\" (UID: \"a9506e6a-0145-4e49-b3f1-acd79fc488f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d"
Oct 30 18:36:54.652831 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.652814    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdmf\" (UniqueName: \"kubernetes.io/projected/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-kube-api-access-ljdmf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.654462 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.654442    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lpp\" (UniqueName: \"kubernetes.io/projected/cb0c4450-cc6e-4124-9177-af902ae797ba-kube-api-access-w7lpp\") pod \"network-check-source-6f78d59f99-8tvw8\" (UID: \"cb0c4450-cc6e-4124-9177-af902ae797ba\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8tvw8"
Oct 30 18:36:54.655157 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.655141    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqnh\" (UniqueName: \"kubernetes.io/projected/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kube-api-access-djqnh\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.656189 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.656169    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfxb\" (UniqueName: \"kubernetes.io/projected/e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb-kube-api-access-bcfxb\") pod \"migrator-66df5d496b-8x557\" (UID: \"e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8x557"
Oct 30 18:36:54.656264 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.656235    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnrh\" (UniqueName: \"kubernetes.io/projected/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-kube-api-access-9hnrh\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.656315 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.656262    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8nq\" (UniqueName: \"kubernetes.io/projected/f5c18994-379c-4961-beac-4a01d127cef0-kube-api-access-8t8nq\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:54.656656 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.656639    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4nw\" (UniqueName: \"kubernetes.io/projected/363e48de-1ba5-4934-a2e7-e38cae066bb8-kube-api-access-nv4nw\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.656937 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.656917    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpc5\" (UniqueName: \"kubernetes.io/projected/708f16f0-f04f-4ea8-8e51-a1d2ca19858a-kube-api-access-tbpc5\") pod \"service-ca-operator-544d749c8c-9m5xn\" (UID: \"708f16f0-f04f-4ea8-8e51-a1d2ca19858a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn"
Oct 30 18:36:54.657080 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.657065    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblvj\" (UniqueName: \"kubernetes.io/projected/200dacf8-0b68-43ea-87ec-e5f67becf695-kube-api-access-cblvj\") pod \"volume-data-source-validator-58c9f5f46d-qc5dv\" (UID: \"200dacf8-0b68-43ea-87ec-e5f67becf695\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qc5dv"
Oct 30 18:36:54.657272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.657258    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qg6m\" (UniqueName: \"kubernetes.io/projected/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-kube-api-access-9qg6m\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.657931 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.657915    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cntt\" (UniqueName: \"kubernetes.io/projected/111c9d09-fd7b-4a3f-99a7-3075467e5e58-kube-api-access-4cntt\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.658074 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.658056    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwm9z\" (UniqueName: \"kubernetes.io/projected/27812221-bc2b-4915-ae7d-105a48913d66-kube-api-access-vwm9z\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.660708 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.660691    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh68w\" (UniqueName: \"kubernetes.io/projected/8d1e8569-d080-44ba-8c4b-b5ae19eaa04c-kube-api-access-rh68w\") pod \"insights-operator-7c4986d449-fs9gh\" (UID: \"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c\") " pod="openshift-insights/insights-operator-7c4986d449-fs9gh"
Oct 30 18:36:54.660852 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.660837    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.661506 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.661489    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.661583 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.661486    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g28g\" (UniqueName: \"kubernetes.io/projected/6914d9ae-4112-4a25-9f37-aa04aa12facc-kube-api-access-6g28g\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.661583 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.661537    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.661583 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.661555    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5p7s\" (UniqueName: \"kubernetes.io/projected/f00a8cdd-93b4-459e-b271-03c501a4c5e5-kube-api-access-s5p7s\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.661954 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.661938    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbqz\" (UniqueName: \"kubernetes.io/projected/49c20dfb-2366-4678-8ffd-9ad49a7774fd-kube-api-access-llbqz\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.662173 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.662158    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7f87\" (UniqueName: \"kubernetes.io/projected/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-api-access-r7f87\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.663133 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.663118    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skln\" (UniqueName: \"kubernetes.io/projected/a395f190-2d86-410f-81ef-93290e75b3a0-kube-api-access-8skln\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.663829 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.663814    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvh28\" (UniqueName: \"kubernetes.io/projected/e844bc94-4bbe-410b-9cb6-807b45c555df-kube-api-access-fvh28\") pod \"kas-connection-checker-685895c7c9-64w7x\" (UID: \"e844bc94-4bbe-410b-9cb6-807b45c555df\") " pod="kube-system/kas-connection-checker-685895c7c9-64w7x"
Oct 30 18:36:54.664666 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.664646    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559c2\" (UniqueName: \"kubernetes.io/projected/eb025a17-a0b0-4f54-9485-b8a85657a54f-kube-api-access-559c2\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.664888 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.664875    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cwn\" (UniqueName: \"kubernetes.io/projected/5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03-kube-api-access-g4cwn\") pod \"console-operator-7cd56b8dbc-spj7q\" (UID: \"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:54.733058 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733031    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733062    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733082    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733097    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733112    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733125    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733145    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733155    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-netns\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733167    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733170    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-cnibin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733173    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-netns\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733141    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-kubelet\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733207    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733220    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-cnibin\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733223    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-systemd\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733238    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733265    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733265    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-sys\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733203    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f00a8cdd-93b4-459e-b271-03c501a4c5e5-host\") pod \"node-ca-96jgv\" (UID: \"f00a8cdd-93b4-459e-b271-03c501a4c5e5\") " pod="openshift-image-registry/node-ca-96jgv"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733292    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733300    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733305    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-sys-fs\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733317    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733338    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a395f190-2d86-410f-81ef-93290e75b3a0-host-slash\") pod \"iptables-alerter-wqk77\" (UID: \"a395f190-2d86-410f-81ef-93290e75b3a0\") " pod="openshift-network-operator/iptables-alerter-wqk77"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733344    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-system-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733339    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733373    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733387    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-systemd-units\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733393    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6914d9ae-4112-4a25-9f37-aa04aa12facc-hosts-file\") pod \"node-resolver-flzfd\" (UID: \"6914d9ae-4112-4a25-9f37-aa04aa12facc\") " pod="openshift-dns/node-resolver-flzfd"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733404    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-multus-certs\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733397    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733445    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733465    4942 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733477    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.733486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733497    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733512    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733527    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.233509127 +0000 UTC m=+1.811604836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : secret "metrics-daemon-secret" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733539    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-kubernetes\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733554    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-lib-modules\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733554    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733584    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-multus\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733569    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-var-lib-kubelet\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733594    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733608    4942 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733618    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733648    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-node-log\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733649    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733660    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.233648505 +0000 UTC m=+1.811744178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733686    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-kubelet\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733699    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-slash\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733702    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733728    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733761    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733770    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733771    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733794    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733811    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733828    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733856    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733862    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-hostroot\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733793    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-run-k8s-cni-cncf-io\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733886    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-ovn\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733894    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733903    4942 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733906    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733912    4942 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dbc964db8-cfqbc: secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733923    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733949    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.733979    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733989    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls podName:327823ad-e078-443a-a05d-70f1bdb181e8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.2339761 +0000 UTC m=+1.812071789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls") pod "image-registry-5dbc964db8-cfqbc" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8") : secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.733992    4942 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734002    4942 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574c7785-ktt9b: secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734000    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-registration-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734030    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-etc-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734034    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734046    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls podName:3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234034999 +0000 UTC m=+1.812130684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls") pod "image-registry-574c7785-ktt9b" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb") : secret "image-registry-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734066    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734084    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734112    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734132    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734150    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734174    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734197    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734194    4942 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734214    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734173    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-bin\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734236    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734238    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-log-socket\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734255    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234243492 +0000 UTC m=+1.812339180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734264    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734283    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-modprobe-d\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734293    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44aaf253-a06d-4d3d-aac8-c1a6424dd260-crio-socket\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734295    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734285    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-dbus\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734338    4942 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734340    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734361    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234354503 +0000 UTC m=+1.812450226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734374    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734385    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-host-cni-netd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734390    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734404    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734422    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734434    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-run-systemd\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734442    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.734414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734459    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734473    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-run\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734475    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734511    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-conf-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734511    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-cni-dir\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734523    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-host\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734536    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234527092 +0000 UTC m=+1.812622796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734526    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734559    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734575    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734612    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/111c9d09-fd7b-4a3f-99a7-3075467e5e58-var-lib-openvswitch\") pod \"ovnkube-node-njg9l\" (UID: \"111c9d09-fd7b-4a3f-99a7-3075467e5e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734619    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-device-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734633    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734647    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49c20dfb-2366-4678-8ffd-9ad49a7774fd-system-cni-dir\") pod \"multus-additional-cni-plugins-bxfx7\" (UID: \"49c20dfb-2366-4678-8ffd-9ad49a7774fd\") " pod="openshift-multus/multus-additional-cni-plugins-bxfx7"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734650    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734679    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734686    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-multus-socket-dir-parent\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734696    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734707    4942 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734712    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734717    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0eb9d479-2705-4c31-84c1-503e24ae3a0e-kubelet-config\") pod \"global-pull-secret-syncer-qw889\" (UID: \"0eb9d479-2705-4c31-84c1-503e24ae3a0e\") " pod="kube-system/global-pull-secret-syncer-qw889"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734728    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734762    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234751939 +0000 UTC m=+1.812847626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734761    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-etc-kubernetes\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734773    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734778    4942 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734786    4942 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734791    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734807    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734819    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234805591 +0000 UTC m=+1.812901262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:54.734846    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.234837005 +0000 UTC m=+1.812932675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734857    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c2d2e58-122c-47f9-8c32-0f73c4e8331c-socket-dir\") pod \"aws-ebs-csi-driver-node-qcpwc\" (UID: \"8c2d2e58-122c-47f9-8c32-0f73c4e8331c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qcpwc"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734845    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysconfig\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734881    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734908    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf92a54-eb83-4b14-a48c-6a1fea2e11d6-etc-sysctl-conf\") pod \"tuned-8z7tf\" (UID: \"bcf92a54-eb83-4b14-a48c-6a1fea2e11d6\") " pod="openshift-cluster-node-tuning-operator/tuned-8z7tf"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734915    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb025a17-a0b0-4f54-9485-b8a85657a54f-host-var-lib-cni-bin\") pod \"multus-94k6t\" (UID: \"eb025a17-a0b0-4f54-9485-b8a85657a54f\") " pod="openshift-multus/multus-94k6t"
Oct 30 18:36:54.736523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.734923    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44aaf253-a06d-4d3d-aac8-c1a6424dd260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:54.813727 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:54.813635    4942 scope.go:122] "RemoveContainer" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"
Oct 30 18:36:55.240057 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.239992    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:55.240057 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240061    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240088    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240108    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240130    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240160    4942 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240215    4942 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240240    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240220109 +0000 UTC m=+2.818315800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : secret "metrics-daemon-secret" not found
Oct 30 18:36:55.240272 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240258    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240247046 +0000 UTC m=+2.818342733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240279    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240309    4942 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240329    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240367    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240351821 +0000 UTC m=+2.818447513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240388    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240435    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240439    4942 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240457    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240501    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240486193 +0000 UTC m=+2.818581885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240508    4942 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240554    4942 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240577    4942 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:55.240589 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240596    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240584087 +0000 UTC m=+2.818679762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:55.240985 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240629    4942 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:55.240985 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240656    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240648482 +0000 UTC m=+2.818744183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:55.240985 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.240694    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.240677372 +0000 UTC m=+2.818773059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:55.240985 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.240980    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363e48de-1ba5-4934-a2e7-e38cae066bb8-service-ca-bundle\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:55.242711 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.242689    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"image-registry-574c7785-ktt9b\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") " pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:55.242792 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.242732    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"image-registry-5dbc964db8-cfqbc\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") " pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:55.412703 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.412666    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:55.421864 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.421844    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:36:55.533323 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.533298    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzfd_6914d9ae-4112-4a25-9f37-aa04aa12facc/dns-node-resolver/0.log"
Oct 30 18:36:55.606456 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.606422    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:36:55.606813 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.606797    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/1.log"
Oct 30 18:36:55.606902 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.606832    4942 generic.go:356] "Generic (PLEG): container finished" podID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03" containerID="df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769" exitCode=255
Oct 30 18:36:55.606954 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.606907    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" event={"ID":"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03","Type":"ContainerDied","Data":"df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769"}
Oct 30 18:36:55.607000 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.606983    4942 scope.go:122] "RemoveContainer" containerID="f83deeb6a79fb3dc01fd90e620ed19b1449496eb5a3525e935fd32ae42b41e0d"
Oct 30 18:36:55.607486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.607466    4942 scope.go:122] "RemoveContainer" containerID="df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769"
Oct 30 18:36:55.607633 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:55.607615    4942 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:36:55.696383 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.696356    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:36:55.699840 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:55.699811    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2eb686_ee8c_42b3_a5b1_8e8b9499e5cb.slice/crio-2662a4415ea5c3844b3a9a3e2b878285814f767639f18badf08523f3959b1e09 WatchSource:0}: Error finding container 2662a4415ea5c3844b3a9a3e2b878285814f767639f18badf08523f3959b1e09: Status 404 returned error can't find the container with id 2662a4415ea5c3844b3a9a3e2b878285814f767639f18badf08523f3959b1e09
Oct 30 18:36:55.725312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:55.725290    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:36:55.730581 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:55.730560    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327823ad_e078_443a_a05d_70f1bdb181e8.slice/crio-74ee1c0f1e585bf8d2f4d96250c5b87481a9e83597f86d8a3ef39c6fe0a3108d WatchSource:0}: Error finding container 74ee1c0f1e585bf8d2f4d96250c5b87481a9e83597f86d8a3ef39c6fe0a3108d: Status 404 returned error can't find the container with id 74ee1c0f1e585bf8d2f4d96250c5b87481a9e83597f86d8a3ef39c6fe0a3108d
Oct 30 18:36:56.123604 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.123537    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96jgv_f00a8cdd-93b4-459e-b271-03c501a4c5e5/node-ca/0.log"
Oct 30 18:36:56.250757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.250715    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:56.250757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.250756    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:56.250959 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.250853    4942 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:56.250959 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.250909    4942 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:56.250959 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.250918    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.250904143 +0000 UTC m=+4.828999813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:56.250959 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.250949    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.250938892 +0000 UTC m=+4.829034566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.250966    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.250986    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.251024    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.251067    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251083    4942 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251107    4942 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251126    4942 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251140    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls podName:27812221-bc2b-4915-ae7d-105a48913d66 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.251125686 +0000 UTC m=+4.829221372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hn5jb" (UID: "27812221-bc2b-4915-ae7d-105a48913d66") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.251087    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251161    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls podName:44aaf253-a06d-4d3d-aac8-c1a6424dd260 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.251151271 +0000 UTC m=+4.829246944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4x55m" (UID: "44aaf253-a06d-4d3d-aac8-c1a6424dd260") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251182    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.251173373 +0000 UTC m=+4.829269083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:56.251192 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251187    4942 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:56.251548 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251221    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.251210524 +0000 UTC m=+4.829306231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:56.251548 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251220    4942 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:56.251548 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.251245    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.25123963 +0000 UTC m=+4.829335335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : secret "metrics-daemon-secret" not found
Oct 30 18:36:56.611646 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.611617    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:36:56.613118 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.613097    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" event={"ID":"327823ad-e078-443a-a05d-70f1bdb181e8","Type":"ContainerStarted","Data":"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"}
Oct 30 18:36:56.613221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.613140    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" event={"ID":"327823ad-e078-443a-a05d-70f1bdb181e8","Type":"ContainerStarted","Data":"74ee1c0f1e585bf8d2f4d96250c5b87481a9e83597f86d8a3ef39c6fe0a3108d"}
Oct 30 18:36:56.614655 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.614631    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c7785-ktt9b" event={"ID":"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb","Type":"ContainerStarted","Data":"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"}
Oct 30 18:36:56.614753 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.614675    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c7785-ktt9b" event={"ID":"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb","Type":"ContainerStarted","Data":"2662a4415ea5c3844b3a9a3e2b878285814f767639f18badf08523f3959b1e09"}
Oct 30 18:36:56.853801 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.853767    4942 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:36:56.854146 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.854132    4942 scope.go:122] "RemoveContainer" containerID="df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769"
Oct 30 18:36:56.854341 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:56.854322    4942 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:36:56.921492 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:56.921415    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cdsk_a09bbce7-7e03-435e-992a-c1d89acdbe5a/serve-healthcheck-canary/0.log"
Oct 30 18:36:57.326288 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:57.326261    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8x557_e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb/migrator/0.log"
Oct 30 18:36:57.618219 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:57.618141    4942 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:57.618580 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:57.618141    4942 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:57.726496 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:57.726462    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-79v5d_a9506e6a-0145-4e49-b3f1-acd79fc488f5/kube-storage-version-migrator-operator/0.log"
Oct 30 18:36:58.269220 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269169    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:36:58.269220 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269225    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269254    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269273    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269339    4942 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269367    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269384    4942 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269397    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert podName:606559f4-9044-4445-90ee-d29a202d2b43 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.26938589 +0000 UTC m=+8.847481560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8pmcl" (UID: "606559f4-9044-4445-90ee-d29a202d2b43") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269338    4942 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269436    4942 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:58.269467 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269451    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.269436688 +0000 UTC m=+8.847532377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269485    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.269472232 +0000 UTC m=+8.847567916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269481    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.269525    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269534    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls podName:bbbfe361-0dd8-4e1e-bf8f-cd48924ec904 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.269519696 +0000 UTC m=+8.847615377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls") pod "dns-default-rc44q" (UID: "bbbfe361-0dd8-4e1e-bf8f-cd48924ec904") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269538    4942 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:58.269844 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:36:58.269582    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs podName:f5c18994-379c-4961-beac-4a01d127cef0 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.269570842 +0000 UTC m=+8.847666514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs") pod "network-metrics-daemon-sfglz" (UID: "f5c18994-379c-4961-beac-4a01d127cef0") : secret "metrics-daemon-secret" not found
Oct 30 18:36:58.271698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.271680    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/27812221-bc2b-4915-ae7d-105a48913d66-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hn5jb\" (UID: \"27812221-bc2b-4915-ae7d-105a48913d66\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:58.271758 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.271696    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44aaf253-a06d-4d3d-aac8-c1a6424dd260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4x55m\" (UID: \"44aaf253-a06d-4d3d-aac8-c1a6424dd260\") " pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:58.411737 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.411716    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"
Oct 30 18:36:58.424494 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.424478    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4x55m"
Oct 30 18:36:58.701441 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.701407    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb"]
Oct 30 18:36:58.704565 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:58.704536    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27812221_bc2b_4915_ae7d_105a48913d66.slice/crio-af78f01354be90eb74e28d261ab82aa1ea1fd68d1e8b29f4a4f7a9446de635a3 WatchSource:0}: Error finding container af78f01354be90eb74e28d261ab82aa1ea1fd68d1e8b29f4a4f7a9446de635a3: Status 404 returned error can't find the container with id af78f01354be90eb74e28d261ab82aa1ea1fd68d1e8b29f4a4f7a9446de635a3
Oct 30 18:36:58.706537 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.706518    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:58.719621 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:58.719599    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4x55m"]
Oct 30 18:36:58.722809 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:36:58.722791    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aaf253_a06d_4d3d_aac8_c1a6424dd260.slice/crio-d548ae101bc9963a3fb370b9fcd315f9f04615d53e2ba30003696dd516dd0da8 WatchSource:0}: Error finding container d548ae101bc9963a3fb370b9fcd315f9f04615d53e2ba30003696dd516dd0da8: Status 404 returned error can't find the container with id d548ae101bc9963a3fb370b9fcd315f9f04615d53e2ba30003696dd516dd0da8
Oct 30 18:36:59.174634 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.174565    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:36:59.228659 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.228358    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:36:59.626605 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.626564    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4x55m" event={"ID":"44aaf253-a06d-4d3d-aac8-c1a6424dd260","Type":"ContainerStarted","Data":"cf7c73024270349a62557cfa57cf655e959a17f6cbc8fc9961fa4744e5f7d947"}
Oct 30 18:36:59.626773 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.626643    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4x55m" event={"ID":"44aaf253-a06d-4d3d-aac8-c1a6424dd260","Type":"ContainerStarted","Data":"641664fcf1f791a22474b5365576e08acb41481f5ad4a11af41336e5c5dc6172"}
Oct 30 18:36:59.626773 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.626677    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4x55m" event={"ID":"44aaf253-a06d-4d3d-aac8-c1a6424dd260","Type":"ContainerStarted","Data":"d548ae101bc9963a3fb370b9fcd315f9f04615d53e2ba30003696dd516dd0da8"}
Oct 30 18:36:59.627903 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:36:59.627876    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb" event={"ID":"27812221-bc2b-4915-ae7d-105a48913d66","Type":"ContainerStarted","Data":"af78f01354be90eb74e28d261ab82aa1ea1fd68d1e8b29f4a4f7a9446de635a3"}
Oct 30 18:37:00.171765 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:00.171733    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:37:00.633366 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:00.633323    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hn5jb" event={"ID":"27812221-bc2b-4915-ae7d-105a48913d66","Type":"ContainerStarted","Data":"82964200f839f6b9b175ef099c7cde0e4d89bfc3a1af217bfda3a704d62a4321"}
Oct 30 18:37:01.638483 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:01.638447    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4x55m" event={"ID":"44aaf253-a06d-4d3d-aac8-c1a6424dd260","Type":"ContainerStarted","Data":"2e2995673ff76b3cdc97f42d2f435a27aff5ae6924426e3ae349dd4928b6b1b9"}
Oct 30 18:37:01.696934 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:01.696907    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:37:01.697276 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:01.697256    4942 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:02.310218 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.310174    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:37:02.310218 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.310227    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:37:02.310478 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.310278    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:37:02.310478 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.310322    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:02.310478 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.310354    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:37:02.310478 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:02.310434    4942 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:37:02.310478 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:02.310467    4942 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:37:02.310723 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:02.310503    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls podName:e8e07000-7744-4dc0-ab28-b877fe1d0cc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:10.31048541 +0000 UTC m=+16.888581081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54gjw" (UID: "e8e07000-7744-4dc0-ab28-b877fe1d0cc8") : secret "samples-operator-tls" not found
Oct 30 18:37:02.310723 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:02.310521    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs podName:363e48de-1ba5-4934-a2e7-e38cae066bb8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:10.310512476 +0000 UTC m=+16.888608150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs") pod "router-default-77bc686d65-bdnkc" (UID: "363e48de-1ba5-4934-a2e7-e38cae066bb8") : secret "router-metrics-certs-default" not found
Oct 30 18:37:02.312745 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.312715    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbbfe361-0dd8-4e1e-bf8f-cd48924ec904-metrics-tls\") pod \"dns-default-rc44q\" (UID: \"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904\") " pod="openshift-dns/dns-default-rc44q"
Oct 30 18:37:02.312745 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.312743    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c18994-379c-4961-beac-4a01d127cef0-metrics-certs\") pod \"network-metrics-daemon-sfglz\" (UID: \"f5c18994-379c-4961-beac-4a01d127cef0\") " pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:37:02.312880 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.312747    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/606559f4-9044-4445-90ee-d29a202d2b43-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8pmcl\" (UID: \"606559f4-9044-4445-90ee-d29a202d2b43\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:37:02.314575 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.314559    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"
Oct 30 18:37:02.319656 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.319639    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfglz"
Oct 30 18:37:02.322801 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.322786    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc44q"
Oct 30 18:37:02.491636 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.491600    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:37:02.555589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.555556    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:37:02.555941 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.555928    4942 scope.go:122] "RemoveContainer" containerID="df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769"
Oct 30 18:37:02.556136 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:02.556118    4942 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-spj7q_openshift-console-operator(5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" podUID="5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03"
Oct 30 18:37:02.606514 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.606449    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8pmcl"]
Oct 30 18:37:02.609604 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:37:02.609567    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606559f4_9044_4445_90ee_d29a202d2b43.slice/crio-f2979c2d1ea4165aba62cc727e5d73ebd9220d52d9c9ed85cb82a207784e455d WatchSource:0}: Error finding container f2979c2d1ea4165aba62cc727e5d73ebd9220d52d9c9ed85cb82a207784e455d: Status 404 returned error can't find the container with id f2979c2d1ea4165aba62cc727e5d73ebd9220d52d9c9ed85cb82a207784e455d
Oct 30 18:37:02.615430 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.615402    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfglz"]
Oct 30 18:37:02.616071 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.616049    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc44q"]
Oct 30 18:37:02.619906 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:37:02.619888    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbfe361_0dd8_4e1e_bf8f_cd48924ec904.slice/crio-4ee14b3675d050b439a3f87aa69a3b4e1801dd0805884f946f56aa2b36665db7 WatchSource:0}: Error finding container 4ee14b3675d050b439a3f87aa69a3b4e1801dd0805884f946f56aa2b36665db7: Status 404 returned error can't find the container with id 4ee14b3675d050b439a3f87aa69a3b4e1801dd0805884f946f56aa2b36665db7
Oct 30 18:37:02.620446 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:37:02.620422    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c18994_379c_4961_beac_4a01d127cef0.slice/crio-92afb787a8eee2e317d85735d033314cff9405f996c05002f4de28e028dccd90 WatchSource:0}: Error finding container 92afb787a8eee2e317d85735d033314cff9405f996c05002f4de28e028dccd90: Status 404 returned error can't find the container with id 92afb787a8eee2e317d85735d033314cff9405f996c05002f4de28e028dccd90
Oct 30 18:37:02.646314 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.646282    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc44q" event={"ID":"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904","Type":"ContainerStarted","Data":"4ee14b3675d050b439a3f87aa69a3b4e1801dd0805884f946f56aa2b36665db7"}
Oct 30 18:37:02.647568 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.647543    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl" event={"ID":"606559f4-9044-4445-90ee-d29a202d2b43","Type":"ContainerStarted","Data":"f2979c2d1ea4165aba62cc727e5d73ebd9220d52d9c9ed85cb82a207784e455d"}
Oct 30 18:37:02.648757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:02.648737    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfglz" event={"ID":"f5c18994-379c-4961-beac-4a01d127cef0","Type":"ContainerStarted","Data":"92afb787a8eee2e317d85735d033314cff9405f996c05002f4de28e028dccd90"}
Oct 30 18:37:04.662771 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.662678    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfglz" event={"ID":"f5c18994-379c-4961-beac-4a01d127cef0","Type":"ContainerStarted","Data":"5a454ce8ee7b47dcb042c7a4c1a8e79691a5f0a8f6b953c58752d46f166a97d2"}
Oct 30 18:37:04.662771 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.662769    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfglz" event={"ID":"f5c18994-379c-4961-beac-4a01d127cef0","Type":"ContainerStarted","Data":"15847cb92fb9dcc521184f50c185f0d38f7145f41bddaf4ef25ee53586786c8a"}
Oct 30 18:37:04.664530 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.664499    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc44q" event={"ID":"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904","Type":"ContainerStarted","Data":"a75a7d3bfbdecaa44e991f3315a952c2ffee6d42f87b2211d7d5ab0f2cd9f750"}
Oct 30 18:37:04.664644 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.664549    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc44q" event={"ID":"bbbfe361-0dd8-4e1e-bf8f-cd48924ec904","Type":"ContainerStarted","Data":"0c41f470c3e25d2990ac0b2d9a0be428a5374533a8d21642487b440bede31a36"}
Oct 30 18:37:04.664644 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.664575    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rc44q"
Oct 30 18:37:04.665898 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:04.665873    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8pmcl" event={"ID":"606559f4-9044-4445-90ee-d29a202d2b43","Type":"ContainerStarted","Data":"e0503012f5b650994d1fc9e45abe8d1485825ac9a676806efbd1af705cbf60e2"}
Oct 30 18:37:06.536633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:06.536602    4942 scope.go:122] "RemoveContainer" containerID="df04c6e8ba90dba92b3bdefbfd74189b1bddc9912296c1b4f572ad67eac7b769"
Oct 30 18:37:06.675289 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:06.675261    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:37:06.675403 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:06.675379    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q" event={"ID":"5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03","Type":"ContainerStarted","Data":"561bfe4e0f8ed2261f3af3021960fe6d29ec3664c2acec2da9a88652f82aa32f"}
Oct 30 18:37:06.675755 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:06.675739    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:37:06.861800 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:06.861744    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-spj7q"
Oct 30 18:37:09.183050 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:09.183000    4942 patch_prober.go:29] interesting pod/image-registry-574c7785-ktt9b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:09.183426 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:09.183065    4942 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-574c7785-ktt9b" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:10.383722 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.383687    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:37:10.384105 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.383781    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:10.386082 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.386056    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8e07000-7744-4dc0-ab28-b877fe1d0cc8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54gjw\" (UID: \"e8e07000-7744-4dc0-ab28-b877fe1d0cc8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:37:10.386188 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.386096    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/363e48de-1ba5-4934-a2e7-e38cae066bb8-metrics-certs\") pod \"router-default-77bc686d65-bdnkc\" (UID: \"363e48de-1ba5-4934-a2e7-e38cae066bb8\") " pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:10.410831 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.410810    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"
Oct 30 18:37:10.413701 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.413682    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:10.456035 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.455707    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5hb6"
Oct 30 18:37:10.702181 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.702155    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw"]
Oct 30 18:37:10.704683 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:10.704658    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77bc686d65-bdnkc"]
Oct 30 18:37:10.708090 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:37:10.708066    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363e48de_1ba5_4934_a2e7_e38cae066bb8.slice/crio-b5e21695d751dc9733163237fd7ad19ef83d34ec9f28488aab11cfc1f50b557e WatchSource:0}: Error finding container b5e21695d751dc9733163237fd7ad19ef83d34ec9f28488aab11cfc1f50b557e: Status 404 returned error can't find the container with id b5e21695d751dc9733163237fd7ad19ef83d34ec9f28488aab11cfc1f50b557e
Oct 30 18:37:11.695551 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:11.695513    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw" event={"ID":"e8e07000-7744-4dc0-ab28-b877fe1d0cc8","Type":"ContainerStarted","Data":"dbbf34ece7479c77fb00722dea622dcf499d1bc1fe2013ff3e04d34c36cb8ea4"}
Oct 30 18:37:11.697143 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:11.697111    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77bc686d65-bdnkc" event={"ID":"363e48de-1ba5-4934-a2e7-e38cae066bb8","Type":"ContainerStarted","Data":"b7e72eebab9fc3a1477baf69a0d570485408e120947cc4a8a099a131108c6b14"}
Oct 30 18:37:11.697301 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:11.697160    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77bc686d65-bdnkc" event={"ID":"363e48de-1ba5-4934-a2e7-e38cae066bb8","Type":"ContainerStarted","Data":"b5e21695d751dc9733163237fd7ad19ef83d34ec9f28488aab11cfc1f50b557e"}
Oct 30 18:37:11.997879 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:11.997800    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:37:12.003706 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.003674    4942 patch_prober.go:29] interesting pod/image-registry-574c7785-ktt9b container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:12.003855 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.003724    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-574c7785-ktt9b" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:12.414530 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.414504    4942 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:12.416903 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.416880    4942 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:12.496162 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.496132    4942 patch_prober.go:29] interesting pod/image-registry-5dbc964db8-cfqbc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:12.496341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.496179    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:12.703425 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.703387    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw" event={"ID":"e8e07000-7744-4dc0-ab28-b877fe1d0cc8","Type":"ContainerStarted","Data":"bcbaab28d77894590cce332f830c10866562d4d5acaf08dcf0d0765994ede12d"}
Oct 30 18:37:12.703886 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.703445    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54gjw" event={"ID":"e8e07000-7744-4dc0-ab28-b877fe1d0cc8","Type":"ContainerStarted","Data":"85c2a385bc099c3d1620bcd7396f72a92dbd40f84557927611f75428d78cc372"}
Oct 30 18:37:12.703886 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.703677    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:12.704916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:12.704895    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77bc686d65-bdnkc"
Oct 30 18:37:14.672257 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:14.672228    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rc44q"
Oct 30 18:37:15.894889 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:15.894855    4942 patch_prober.go:29] interesting pod/image-registry-5dbc964db8-cfqbc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:15.895272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:15.894914    4942 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:18.852843 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.852812    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bfz8b"]
Oct 30 18:37:18.857816 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.857797    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.928924 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.928900    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:37:18.929111 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.928946    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:37:18.929403 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.929390    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:37:18.929403 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.929398    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vs59c\""
Oct 30 18:37:18.929520 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.929459    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:37:18.952784 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952753    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-metrics-client-ca\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.952895 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952801    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-root\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.952895 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952864    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.952895 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952891    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-sys\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.953030 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952905    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-tls\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.953030 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.952979    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkww\" (UniqueName: \"kubernetes.io/projected/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-kube-api-access-phkww\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.953107 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.953030    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-textfile\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.953107 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.953082    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:18.953107 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:18.953103    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-wtmp\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054237 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054210    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-sys\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054242    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-tls\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054264    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkww\" (UniqueName: \"kubernetes.io/projected/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-kube-api-access-phkww\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054291    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-textfile\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054316    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-sys\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054480 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054402    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054480 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054437    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-wtmp\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054554 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054495    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-metrics-client-ca\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054554 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054542    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-root\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054572    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054624    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-textfile\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054634    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-wtmp\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.054653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054637    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-root\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.055022 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.054984    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-metrics-client-ca\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.055109 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.055046    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.056722 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.056703    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-tls\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.056802 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.056726    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.067275 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.067254    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkww\" (UniqueName: \"kubernetes.io/projected/b8ad0765-0cf3-4882-9ced-c0f6280f3ca9-kube-api-access-phkww\") pod \"node-exporter-bfz8b\" (UID: \"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9\") " pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.173587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.173533    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bfz8b"
Oct 30 18:37:19.185264 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:37:19.185238    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ad0765_0cf3_4882_9ced_c0f6280f3ca9.slice/crio-2e49003272c29fd1335f785211b47ef0d6d1297fa64c635d3714c35f183ce969 WatchSource:0}: Error finding container 2e49003272c29fd1335f785211b47ef0d6d1297fa64c635d3714c35f183ce969: Status 404 returned error can't find the container with id 2e49003272c29fd1335f785211b47ef0d6d1297fa64c635d3714c35f183ce969
Oct 30 18:37:19.737489 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.737452    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bfz8b" event={"ID":"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9","Type":"ContainerStarted","Data":"2e49003272c29fd1335f785211b47ef0d6d1297fa64c635d3714c35f183ce969"}
Oct 30 18:37:19.970456 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:19.970428    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg9l"
Oct 30 18:37:20.744028 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:20.743971    4942 generic.go:356] "Generic (PLEG): container finished" podID="b8ad0765-0cf3-4882-9ced-c0f6280f3ca9" containerID="c1baa6a7c9cf10eb33c186ea71f75c4bb6e52bf17b98c645f9bf6e9a083707a3" exitCode=0
Oct 30 18:37:20.744196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:20.744062    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bfz8b" event={"ID":"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9","Type":"ContainerDied","Data":"c1baa6a7c9cf10eb33c186ea71f75c4bb6e52bf17b98c645f9bf6e9a083707a3"}
Oct 30 18:37:21.750330 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:21.750292    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bfz8b" event={"ID":"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9","Type":"ContainerStarted","Data":"4a5385676e1ab2fc65eb31ca333503b863164824fca3f41c1d1f28dbd160d418"}
Oct 30 18:37:21.750692 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:21.750352    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bfz8b" event={"ID":"b8ad0765-0cf3-4882-9ced-c0f6280f3ca9","Type":"ContainerStarted","Data":"99cf7463e1d2e691eaab3181b4abeec8c21d3120dc5938c097f676c670e6058f"}
Oct 30 18:37:21.783128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:21.783076    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bfz8b" podStartSLOduration=2.719687044 podStartE2EDuration="3.783064493s" totalImagesPullingTime="997.917195ms" totalInitContainerRuntime="65.460254ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:21.780708375 +0000 UTC m=+28.358804093" watchObservedRunningTime="2026-10-30 18:37:21.783064493 +0000 UTC m=+28.361160185"
Oct 30 18:37:22.002873 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:22.002801    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:37:22.496271 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:22.496241    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:37:26.725448 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:26.725409    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:37:29.231768 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:29.231736    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lz7bk"
Oct 30 18:37:33.984386 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:33.981703    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:37:37.024868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.024809    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-574c7785-ktt9b" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry" containerID="cri-o://572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849" gracePeriod=30
Oct 30 18:37:37.265049 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.264997    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:37:37.300201 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300135    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300201 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300189    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300215    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300339    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300482 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300389    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300482 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300441    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300482 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300469    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.300656 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300507    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") pod \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\" (UID: \"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb\") "
Oct 30 18:37:37.301071 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.300916    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:37.302182 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.302141    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:37.303469 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.303438    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:37.303775 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.303749    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:37.303919 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.303887    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:37.304501 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.304467    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "kube-api-access-m8jxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:37.304750 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.304733    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:37.309725 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.309698    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted" pod "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" (UID: "3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:37.401982 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.401944    4942 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-installation-pull-secrets\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.401982 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.401979    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jxk\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-kube-api-access-m8jxk\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.401994    4942 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-image-registry-private-configuration\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.402023    4942 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.402037    4942 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-trusted-ca\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.402045    4942 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-bound-sa-token\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.402054    4942 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-registry-certificates\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.402221 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.402063    4942 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb-ca-trust-extracted\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:37.818313 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.818282    4942 generic.go:356] "Generic (PLEG): container finished" podID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerID="572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849" exitCode=0
Oct 30 18:37:37.818483 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.818336    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c7785-ktt9b"
Oct 30 18:37:37.818483 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.818376    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c7785-ktt9b" event={"ID":"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb","Type":"ContainerDied","Data":"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"}
Oct 30 18:37:37.818483 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.818431    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c7785-ktt9b" event={"ID":"3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb","Type":"ContainerDied","Data":"2662a4415ea5c3844b3a9a3e2b878285814f767639f18badf08523f3959b1e09"}
Oct 30 18:37:37.818483 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.818452    4942 scope.go:122] "RemoveContainer" containerID="572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"
Oct 30 18:37:37.826702 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.826687    4942 scope.go:122] "RemoveContainer" containerID="572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"
Oct 30 18:37:37.826969 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:37.826949    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849\": container with ID starting with 572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849 not found: ID does not exist" containerID="572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"
Oct 30 18:37:37.827069 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.826978    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849"} err="failed to get container status \"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849\": rpc error: code = NotFound desc = could not find container \"572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849\": container with ID starting with 572d08a545ed4bf025fd4951b52a207f983612e5a834dbc3cb5ccbdc5c0f9849 not found: ID does not exist"
Oct 30 18:37:37.843231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.843207    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:37:37.851879 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:37.851858    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-574c7785-ktt9b"]
Oct 30 18:37:39.546155 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:39.546109    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" path="/var/lib/kubelet/pods/3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb/volumes"
Oct 30 18:37:56.726285 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:56.726242    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:37:58.912867 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:58.912833    4942 generic.go:356] "Generic (PLEG): container finished" podID="a9506e6a-0145-4e49-b3f1-acd79fc488f5" containerID="9ed675487e070ba2ad2987c8b7f2ab44d69fe921cedd18f0fbe27d675ce52a52" exitCode=0
Oct 30 18:37:58.913260 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:58.912902    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d" event={"ID":"a9506e6a-0145-4e49-b3f1-acd79fc488f5","Type":"ContainerDied","Data":"9ed675487e070ba2ad2987c8b7f2ab44d69fe921cedd18f0fbe27d675ce52a52"}
Oct 30 18:37:58.913306 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:58.913289    4942 scope.go:122] "RemoveContainer" containerID="9ed675487e070ba2ad2987c8b7f2ab44d69fe921cedd18f0fbe27d675ce52a52"
Oct 30 18:37:59.003942 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.003848    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry" containerID="cri-o://e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b" gracePeriod=30
Oct 30 18:37:59.258185 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.258132    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:37:59.366616 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366583    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366770 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366649    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366770 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366688    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366770 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366721    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366780    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366840    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.366963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.366877    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.367211 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.367059    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") pod \"327823ad-e078-443a-a05d-70f1bdb181e8\" (UID: \"327823ad-e078-443a-a05d-70f1bdb181e8\") "
Oct 30 18:37:59.367603 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.367325    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:59.367603 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.367468    4942 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-trusted-ca\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.367940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.367894    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:59.369438 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.369391    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:59.369438 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.369412    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:59.370091 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.370064    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:59.370433 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.370396    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "kube-api-access-trnkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:59.370969 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.370938    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:59.374779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.374746    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted" pod "327823ad-e078-443a-a05d-70f1bdb181e8" (UID: "327823ad-e078-443a-a05d-70f1bdb181e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:59.468548 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468516    4942 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-bound-sa-token\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468548 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468548    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnkl\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-kube-api-access-trnkl\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468560    4942 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-installation-pull-secrets\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468571    4942 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/327823ad-e078-443a-a05d-70f1bdb181e8-image-registry-private-configuration\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468581    4942 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/327823ad-e078-443a-a05d-70f1bdb181e8-registry-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468590    4942 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/327823ad-e078-443a-a05d-70f1bdb181e8-registry-certificates\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.468713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.468599    4942 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/327823ad-e078-443a-a05d-70f1bdb181e8-ca-trust-extracted\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.918762 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.918683    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-79v5d" event={"ID":"a9506e6a-0145-4e49-b3f1-acd79fc488f5","Type":"ContainerStarted","Data":"ffb8c409e1c2b9eb2208050558a22449e32b5718f26289b7b0efd0b0877f5c02"}
Oct 30 18:37:59.919969 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.919945    4942 generic.go:356] "Generic (PLEG): container finished" podID="327823ad-e078-443a-a05d-70f1bdb181e8" containerID="e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b" exitCode=0
Oct 30 18:37:59.920098 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.919977    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc"
Oct 30 18:37:59.920098 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.919991    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" event={"ID":"327823ad-e078-443a-a05d-70f1bdb181e8","Type":"ContainerDied","Data":"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"}
Oct 30 18:37:59.920098 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.920047    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc964db8-cfqbc" event={"ID":"327823ad-e078-443a-a05d-70f1bdb181e8","Type":"ContainerDied","Data":"74ee1c0f1e585bf8d2f4d96250c5b87481a9e83597f86d8a3ef39c6fe0a3108d"}
Oct 30 18:37:59.920098 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.920066    4942 scope.go:122] "RemoveContainer" containerID="e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"
Oct 30 18:37:59.927880 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.927854    4942 scope.go:122] "RemoveContainer" containerID="e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"
Oct 30 18:37:59.928193 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:37:59.928174    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b\": container with ID starting with e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b not found: ID does not exist" containerID="e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"
Oct 30 18:37:59.928273 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.928203    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b"} err="failed to get container status \"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b\": rpc error: code = NotFound desc = could not find container \"e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b\": container with ID starting with e45ca21298c0b271ef74dd4998fd5deab95b2d2c51a351f62e902be5ffac310b not found: ID does not exist"
Oct 30 18:37:59.971806 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.971775    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:37:59.975188 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:37:59.975168    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dbc964db8-cfqbc"]
Oct 30 18:38:01.541677 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:01.541639    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" path="/var/lib/kubelet/pods/327823ad-e078-443a-a05d-70f1bdb181e8/volumes"
Oct 30 18:38:03.938998 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:03.938966    4942 generic.go:356] "Generic (PLEG): container finished" podID="8d1e8569-d080-44ba-8c4b-b5ae19eaa04c" containerID="41ce3c148b328acf8d23045b751d824337b8aa2da224533e392b7adb5ceb2faf" exitCode=0
Oct 30 18:38:03.939380 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:03.939029    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fs9gh" event={"ID":"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c","Type":"ContainerDied","Data":"41ce3c148b328acf8d23045b751d824337b8aa2da224533e392b7adb5ceb2faf"}
Oct 30 18:38:03.939380 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:03.939278    4942 scope.go:122] "RemoveContainer" containerID="41ce3c148b328acf8d23045b751d824337b8aa2da224533e392b7adb5ceb2faf"
Oct 30 18:38:04.945271 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:04.945235    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fs9gh" event={"ID":"8d1e8569-d080-44ba-8c4b-b5ae19eaa04c","Type":"ContainerStarted","Data":"d76807b042895e707546e213ecc0a47d42e6d16f4c317bdd0d9b8b1df87e65e4"}
Oct 30 18:38:09.965616 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:09.965582    4942 generic.go:356] "Generic (PLEG): container finished" podID="708f16f0-f04f-4ea8-8e51-a1d2ca19858a" containerID="8107b3c3fc53a8331e0950b4c6f1fe8e8ebc75f4221e94f766b227a7627161c7" exitCode=0
Oct 30 18:38:09.966152 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:09.965656    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn" event={"ID":"708f16f0-f04f-4ea8-8e51-a1d2ca19858a","Type":"ContainerDied","Data":"8107b3c3fc53a8331e0950b4c6f1fe8e8ebc75f4221e94f766b227a7627161c7"}
Oct 30 18:38:09.966152 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:09.966051    4942 scope.go:122] "RemoveContainer" containerID="8107b3c3fc53a8331e0950b4c6f1fe8e8ebc75f4221e94f766b227a7627161c7"
Oct 30 18:38:10.970752 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:38:10.970719    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9m5xn" event={"ID":"708f16f0-f04f-4ea8-8e51-a1d2ca19858a","Type":"ContainerStarted","Data":"0391f8f044261f3d38edea3d81d509dd64f0ab6390aa57a7b7d54cac201ea345"}
Oct 30 18:38:26.723746 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:38:26.723708    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:38:56.727276 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:38:56.727232    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:39:26.725935 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:39:26.725877    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:39:56.723889 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:39:56.723841    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:40:26.724095 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:40:26.724044    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:40:56.728250 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:40:56.728158    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:41:26.724962 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:41:26.724922    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:41:53.558981 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:41:53.558949    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:41:53.561137 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:41:53.561112    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:41:56.724337 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:41:56.724298    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:42:26.723707 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:42:26.723668    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:42:28.154448 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154409    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"]
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154722    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry"
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154733    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry"
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154744    4942 cpu_manager.go:414] "containerMap: removing container" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry"
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154750    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry"
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154831    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2eb686-ee8c-42b3-a5b1-8e8b9499e5cb" containerName="registry"
Oct 30 18:42:28.154849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.154841    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="327823ad-e078-443a-a05d-70f1bdb181e8" containerName="registry"
Oct 30 18:42:28.157664 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.157648    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.169408 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.169382    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 18:42:28.169511 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.169405    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 18:42:28.170354 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.170334    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"]
Oct 30 18:42:28.170818 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.170798    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 18:42:28.171020 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.170975    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 18:42:28.171100 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.171039    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 18:42:28.171100 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.171065    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-sf4sk\""
Oct 30 18:42:28.206904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.206882    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.207023 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.206916    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/296c6249-fb28-490f-a08d-c0eb4563ee7c-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.207023 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.206935    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cdt\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-kube-api-access-h7cdt\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.307338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.307315    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.307426 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.307357    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/296c6249-fb28-490f-a08d-c0eb4563ee7c-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.307426 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.307382    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cdt\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-kube-api-access-h7cdt\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.307702 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.307682    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/296c6249-fb28-490f-a08d-c0eb4563ee7c-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.309870 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.309850    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.320299 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.320264    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cdt\" (UniqueName: \"kubernetes.io/projected/296c6249-fb28-490f-a08d-c0eb4563ee7c-kube-api-access-h7cdt\") pod \"keda-metrics-apiserver-86bd856cfb-jpwzr\" (UID: \"296c6249-fb28-490f-a08d-c0eb4563ee7c\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.353466 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.353442    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-v4g42"]
Oct 30 18:42:28.356621 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.356605    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.375244 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.375218    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 18:42:28.375513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.375493    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-v4g42"]
Oct 30 18:42:28.408178 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.408114    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-certificates\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.408178 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.408155    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlb6l\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-kube-api-access-zlb6l\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.470417 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.470387    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:28.508821 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.508789    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-certificates\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.508964 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.508838    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlb6l\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-kube-api-access-zlb6l\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.511246 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.511222    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-certificates\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.525051 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.525022    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlb6l\" (UniqueName: \"kubernetes.io/projected/adabdfed-5b0e-4228-a31a-3ad2663d82d0-kube-api-access-zlb6l\") pod \"keda-admission-7b7c985c58-v4g42\" (UID: \"adabdfed-5b0e-4228-a31a-3ad2663d82d0\") " pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.668337 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.668261    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:28.748450 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.748419    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"]
Oct 30 18:42:28.751374 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:42:28.751345    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296c6249_fb28_490f_a08d_c0eb4563ee7c.slice/crio-97d1f2105cbbaf2da9fa0f026bc3f4326d2c0156ff70e9a2f8f9085882fcc321 WatchSource:0}: Error finding container 97d1f2105cbbaf2da9fa0f026bc3f4326d2c0156ff70e9a2f8f9085882fcc321: Status 404 returned error can't find the container with id 97d1f2105cbbaf2da9fa0f026bc3f4326d2c0156ff70e9a2f8f9085882fcc321
Oct 30 18:42:28.752791 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.752775    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:42:28.925900 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.925828    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr" event={"ID":"296c6249-fb28-490f-a08d-c0eb4563ee7c","Type":"ContainerStarted","Data":"97d1f2105cbbaf2da9fa0f026bc3f4326d2c0156ff70e9a2f8f9085882fcc321"}
Oct 30 18:42:28.987929 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:28.987900    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-v4g42"]
Oct 30 18:42:28.990883 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:42:28.990858    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadabdfed_5b0e_4228_a31a_3ad2663d82d0.slice/crio-e30350cddd5edf2013c1be1085faf6d59845e759c6b6fa592b9d2b16d88971c8 WatchSource:0}: Error finding container e30350cddd5edf2013c1be1085faf6d59845e759c6b6fa592b9d2b16d88971c8: Status 404 returned error can't find the container with id e30350cddd5edf2013c1be1085faf6d59845e759c6b6fa592b9d2b16d88971c8
Oct 30 18:42:29.932795 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:29.932746    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-v4g42" event={"ID":"adabdfed-5b0e-4228-a31a-3ad2663d82d0","Type":"ContainerStarted","Data":"e30350cddd5edf2013c1be1085faf6d59845e759c6b6fa592b9d2b16d88971c8"}
Oct 30 18:42:32.947880 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.947827    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr" event={"ID":"296c6249-fb28-490f-a08d-c0eb4563ee7c","Type":"ContainerStarted","Data":"57490c0af7e7a631892e43d2fce45ae5d09035a0669c76158625f8d50bc8b132"}
Oct 30 18:42:32.948395 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.948041    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:32.949565 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.949538    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-v4g42" event={"ID":"adabdfed-5b0e-4228-a31a-3ad2663d82d0","Type":"ContainerStarted","Data":"e551d542b829c45143e5c51ff6a28cc73bddf97a2c31bad79f37da600f0d469b"}
Oct 30 18:42:32.949677 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.949665    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:32.971619 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.971572    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr" podStartSLOduration=1.252207276 podStartE2EDuration="4.971555756s" totalImagesPullingTime="3.71934848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:32.970400464 +0000 UTC m=+339.548496155" watchObservedRunningTime="2026-10-30 18:42:32.971555756 +0000 UTC m=+339.549651450"
Oct 30 18:42:32.994307 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:32.994257    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-v4g42" podStartSLOduration=1.620485195 podStartE2EDuration="4.994239501s" totalImagesPullingTime="3.373754306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:32.9931905 +0000 UTC m=+339.571286191" watchObservedRunningTime="2026-10-30 18:42:32.994239501 +0000 UTC m=+339.572335196"
Oct 30 18:42:43.958619 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:43.958585    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-jpwzr"
Oct 30 18:42:53.955863 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:42:53.955831    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-v4g42"
Oct 30 18:42:56.724523 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:42:56.724486    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:43:26.727611 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:43:26.727564    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:43:56.270811 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.270772    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-hz5fg"]
Oct 30 18:43:56.274244 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.274220    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.285032 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.284991    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-hz5fg"]
Oct 30 18:43:56.306064 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.305998    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwmr\" (UniqueName: \"kubernetes.io/projected/9af38e72-3b1d-4b7d-b581-0565244a8706-kube-api-access-hbwmr\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.306252 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.306123    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9af38e72-3b1d-4b7d-b581-0565244a8706-data\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.319168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.319134    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 18:43:56.319327 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.319173    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x652p\""
Oct 30 18:43:56.381380 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.381346    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:43:56.381380 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.381346    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:43:56.406970 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.406933    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwmr\" (UniqueName: \"kubernetes.io/projected/9af38e72-3b1d-4b7d-b581-0565244a8706-kube-api-access-hbwmr\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.407127 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.406982    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9af38e72-3b1d-4b7d-b581-0565244a8706-data\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.407410 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.407385    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9af38e72-3b1d-4b7d-b581-0565244a8706-data\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.420739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.420708    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwmr\" (UniqueName: \"kubernetes.io/projected/9af38e72-3b1d-4b7d-b581-0565244a8706-kube-api-access-hbwmr\") pod \"seaweedfs-78bb9c6ff8-hz5fg\" (UID: \"9af38e72-3b1d-4b7d-b581-0565244a8706\") " pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.585755 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.585665    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:43:56.723763 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:43:56.723726    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:43:56.859601 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:56.859535    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-hz5fg"]
Oct 30 18:43:56.863035 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:43:56.862993    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af38e72_3b1d_4b7d_b581_0565244a8706.slice/crio-8e44d5facfafa32266ecb6b21ab629fd058081dac1617303323d16a5cdda4a97 WatchSource:0}: Error finding container 8e44d5facfafa32266ecb6b21ab629fd058081dac1617303323d16a5cdda4a97: Status 404 returned error can't find the container with id 8e44d5facfafa32266ecb6b21ab629fd058081dac1617303323d16a5cdda4a97
Oct 30 18:43:57.293460 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:43:57.293422    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg" event={"ID":"9af38e72-3b1d-4b7d-b581-0565244a8706","Type":"ContainerStarted","Data":"8e44d5facfafa32266ecb6b21ab629fd058081dac1617303323d16a5cdda4a97"}
Oct 30 18:44:00.308175 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:44:00.308121    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg" event={"ID":"9af38e72-3b1d-4b7d-b581-0565244a8706","Type":"ContainerStarted","Data":"6cf897da3db156b66936c9fe57f80c853d8cf5c28427fc24241876edfb1feb91"}
Oct 30 18:44:00.308688 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:44:00.308281    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:44:00.331310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:44:00.331239    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg" podStartSLOduration=1.668719842 podStartE2EDuration="4.331222347s" totalImagesPullingTime="2.662502505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:00.329990361 +0000 UTC m=+426.908086054" watchObservedRunningTime="2026-10-30 18:44:00.331222347 +0000 UTC m=+426.909318040"
Oct 30 18:44:06.315445 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:44:06.315411    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-hz5fg"
Oct 30 18:44:26.724708 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:44:26.724667    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:44:56.724694 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:44:56.724654    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:45:23.655476 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.655388    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-85lmc"]
Oct 30 18:45:23.658741 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.658714    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.666778 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.666750    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 18:45:23.666889 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.666784    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-flvbg\""
Oct 30 18:45:23.673310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.673272    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-85lmc"]
Oct 30 18:45:23.723562 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.723506    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-wzhdb"]
Oct 30 18:45:23.727166 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.727137    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.744948 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.744913    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-wzhdb"]
Oct 30 18:45:23.769434 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.769403    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:45:23.769647 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.769492    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-szdm9\""
Oct 30 18:45:23.769647 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.769580    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:45:23.803693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.803655    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e72c297-077d-4ddd-9c97-895a84666e34-tls-certs\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.803851 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.803710    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-cert\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.803851 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.803803    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzml\" (UniqueName: \"kubernetes.io/projected/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-kube-api-access-nbzml\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.803851 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.803846    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.803978 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.803915    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8sl9\" (UniqueName: \"kubernetes.io/projected/7e72c297-077d-4ddd-9c97-895a84666e34-kube-api-access-c8sl9\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.905207 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.905151    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e72c297-077d-4ddd-9c97-895a84666e34-tls-certs\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.905207 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.905212    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-cert\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.905503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.905242    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzml\" (UniqueName: \"kubernetes.io/projected/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-kube-api-access-nbzml\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.905503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.905264    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.905503 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.905289    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8sl9\" (UniqueName: \"kubernetes.io/projected/7e72c297-077d-4ddd-9c97-895a84666e34-kube-api-access-c8sl9\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.907884 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.907852    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-cert\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.907979 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.907942    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e72c297-077d-4ddd-9c97-895a84666e34-tls-certs\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.908048 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.907942    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.919325 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.919297    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzml\" (UniqueName: \"kubernetes.io/projected/adbf527c-4b4a-4ffe-b22f-97ecd5c320b8-kube-api-access-nbzml\") pod \"odh-model-controller-7f9f8cdc65-wzhdb\" (UID: \"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8\") " pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:23.922884 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.922860    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8sl9\" (UniqueName: \"kubernetes.io/projected/7e72c297-077d-4ddd-9c97-895a84666e34-kube-api-access-c8sl9\") pod \"model-serving-api-56ddc4c448-85lmc\" (UID: \"7e72c297-077d-4ddd-9c97-895a84666e34\") " pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:23.973330 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:23.973291    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:24.041549 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:24.041506    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:24.254258 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:24.254217    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-85lmc"]
Oct 30 18:45:24.257591 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:45:24.257550    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e72c297_077d_4ddd_9c97_895a84666e34.slice/crio-6a22812215cfb8e8752e096c15284593cb81d07d7df874a111db1a7f4469c6ff WatchSource:0}: Error finding container 6a22812215cfb8e8752e096c15284593cb81d07d7df874a111db1a7f4469c6ff: Status 404 returned error can't find the container with id 6a22812215cfb8e8752e096c15284593cb81d07d7df874a111db1a7f4469c6ff
Oct 30 18:45:24.325560 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:24.325519    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-wzhdb"]
Oct 30 18:45:24.329109 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:45:24.329079    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbf527c_4b4a_4ffe_b22f_97ecd5c320b8.slice/crio-11adc85171674606df0238f6a524f8f7b13b66a0a8dec78617c25dd48ff20923 WatchSource:0}: Error finding container 11adc85171674606df0238f6a524f8f7b13b66a0a8dec78617c25dd48ff20923: Status 404 returned error can't find the container with id 11adc85171674606df0238f6a524f8f7b13b66a0a8dec78617c25dd48ff20923
Oct 30 18:45:24.651682 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:24.651587    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb" event={"ID":"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8","Type":"ContainerStarted","Data":"11adc85171674606df0238f6a524f8f7b13b66a0a8dec78617c25dd48ff20923"}
Oct 30 18:45:24.653145 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:24.653102    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-85lmc" event={"ID":"7e72c297-077d-4ddd-9c97-895a84666e34","Type":"ContainerStarted","Data":"6a22812215cfb8e8752e096c15284593cb81d07d7df874a111db1a7f4469c6ff"}
Oct 30 18:45:26.734262 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:45:26.734168    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:45:27.670409 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.670370    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb" event={"ID":"adbf527c-4b4a-4ffe-b22f-97ecd5c320b8","Type":"ContainerStarted","Data":"c74773d0d09a70b6cc339bdbc0f5a969247f83d3b1e8101e1b4f49e5309ff429"}
Oct 30 18:45:27.670928 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.670487    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:27.672111 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.672087    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-85lmc" event={"ID":"7e72c297-077d-4ddd-9c97-895a84666e34","Type":"ContainerStarted","Data":"7f5d40860948aedbea452bec6f773e9d1350b894fd657c5ff389e55065e69206"}
Oct 30 18:45:27.672193 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.672144    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:27.692415 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.692348    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb" podStartSLOduration=1.8663169499999999 podStartE2EDuration="4.692333714s" totalImagesPullingTime="2.826016764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.69059825 +0000 UTC m=+514.268693942" watchObservedRunningTime="2026-10-30 18:45:27.692333714 +0000 UTC m=+514.270429406"
Oct 30 18:45:27.764164 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:27.764106    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-85lmc" podStartSLOduration=2.000525922 podStartE2EDuration="4.764091357s" totalImagesPullingTime="2.763565435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.763790093 +0000 UTC m=+514.341885788" watchObservedRunningTime="2026-10-30 18:45:27.764091357 +0000 UTC m=+514.342187049"
Oct 30 18:45:38.677836 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:38.677802    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-wzhdb"
Oct 30 18:45:38.680574 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:45:38.680551    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-85lmc"
Oct 30 18:45:56.724515 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:45:56.724474    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:46:26.725442 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:46:26.725330    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:46:53.584070 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:46:53.583975    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:46:53.585197 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:46:53.585172    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:46:56.725502 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:46:56.725467    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:47:05.624628 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.624595    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:05.627920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.627899    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.635200 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.635166    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 30 18:47:05.636888 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.636861    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:05.695028 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.694982    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.695134 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.695050    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxpn\" (UniqueName: \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.796121 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.796090    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.796255 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.796133    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxpn\" (UniqueName: \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.796496 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.796473    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.809168 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.809145    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxpn\" (UniqueName: \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\") pod \"seaweedfs-tls-custom-64cc6669cb-kt2k2\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:05.940533 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:05.940504    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:06.217820 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:06.217751    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:06.221813 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:47:06.221790    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49eba81_91bb_4472_af81_fef0cf06604e.slice/crio-270fbde0519e6587c53e75fe52e4226e072551cd3763c00ea85b66e7b6f6072c WatchSource:0}: Error finding container 270fbde0519e6587c53e75fe52e4226e072551cd3763c00ea85b66e7b6f6072c: Status 404 returned error can't find the container with id 270fbde0519e6587c53e75fe52e4226e072551cd3763c00ea85b66e7b6f6072c
Oct 30 18:47:07.070816 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:07.070785    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" event={"ID":"c49eba81-91bb-4472-af81-fef0cf06604e","Type":"ContainerStarted","Data":"5e6399b8ad7e94b707d12780a5165615aa5bfba76706199a6f13a2afaf455412"}
Oct 30 18:47:07.071257 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:07.070838    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" event={"ID":"c49eba81-91bb-4472-af81-fef0cf06604e","Type":"ContainerStarted","Data":"270fbde0519e6587c53e75fe52e4226e072551cd3763c00ea85b66e7b6f6072c"}
Oct 30 18:47:07.089602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:07.089554    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" podStartSLOduration=1.773942591 podStartE2EDuration="2.089541512s" totalImagesPullingTime="315.598921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:07.089273018 +0000 UTC m=+613.667368715" watchObservedRunningTime="2026-10-30 18:47:07.089541512 +0000 UTC m=+613.667637205"
Oct 30 18:47:08.468686 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:08.468647    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:09.081344 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:09.081296    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" podUID="c49eba81-91bb-4472-af81-fef0cf06604e" containerName="seaweedfs-tls-custom" containerID="cri-o://5e6399b8ad7e94b707d12780a5165615aa5bfba76706199a6f13a2afaf455412" gracePeriod=30
Oct 30 18:47:26.725158 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:47:26.725119    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:47:37.191263 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.191230    4942 generic.go:356] "Generic (PLEG): container finished" podID="c49eba81-91bb-4472-af81-fef0cf06604e" containerID="5e6399b8ad7e94b707d12780a5165615aa5bfba76706199a6f13a2afaf455412" exitCode=0
Oct 30 18:47:37.191668 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.191305    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" event={"ID":"c49eba81-91bb-4472-af81-fef0cf06604e","Type":"ContainerDied","Data":"5e6399b8ad7e94b707d12780a5165615aa5bfba76706199a6f13a2afaf455412"}
Oct 30 18:47:37.239505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.239475    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:37.340517 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.340424    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\" (UniqueName: \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\") pod \"c49eba81-91bb-4472-af81-fef0cf06604e\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") "
Oct 30 18:47:37.340517 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.340497    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\" (UniqueName: \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\") pod \"c49eba81-91bb-4472-af81-fef0cf06604e\" (UID: \"c49eba81-91bb-4472-af81-fef0cf06604e\") "
Oct 30 18:47:37.341922 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.341890    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data" pod "c49eba81-91bb-4472-af81-fef0cf06604e" (UID: "c49eba81-91bb-4472-af81-fef0cf06604e"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:37.342767 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.342739    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn" pod "c49eba81-91bb-4472-af81-fef0cf06604e" (UID: "c49eba81-91bb-4472-af81-fef0cf06604e"). InnerVolumeSpecName "kube-api-access-qbxpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:37.373520 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.373479    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"]
Oct 30 18:47:37.373848 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.373835    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c49eba81-91bb-4472-af81-fef0cf06604e" containerName="seaweedfs-tls-custom"
Oct 30 18:47:37.373891 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.373850    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49eba81-91bb-4472-af81-fef0cf06604e" containerName="seaweedfs-tls-custom"
Oct 30 18:47:37.373929 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.373924    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49eba81-91bb-4472-af81-fef0cf06604e" containerName="seaweedfs-tls-custom"
Oct 30 18:47:37.378164 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.378141    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.399999 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.399959    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Oct 30 18:47:37.406371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.406338    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"]
Oct 30 18:47:37.441397 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.441350    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.441589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.441409    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80115876-a6e1-4f39-a6c9-6847373e70f9-data\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.441589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.441565    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7rb\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-kube-api-access-bt7rb\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.441673 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.441619    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxpn\" (UniqueName: \"kubernetes.io/projected/c49eba81-91bb-4472-af81-fef0cf06604e-kube-api-access-qbxpn\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:37.441673 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.441634    4942 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c49eba81-91bb-4472-af81-fef0cf06604e-data\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:37.542026 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.541984    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7rb\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-kube-api-access-bt7rb\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.542199 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.542060    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.542199 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.542100    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80115876-a6e1-4f39-a6c9-6847373e70f9-data\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.542529 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.542509    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/80115876-a6e1-4f39-a6c9-6847373e70f9-data\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.544783 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.544750    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.556096 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.556069    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7rb\" (UniqueName: \"kubernetes.io/projected/80115876-a6e1-4f39-a6c9-6847373e70f9-kube-api-access-bt7rb\") pod \"seaweedfs-tls-custom-685c9f986c-qpf7p\" (UID: \"80115876-a6e1-4f39-a6c9-6847373e70f9\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.690866 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.690831    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"
Oct 30 18:47:37.970955 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.970866    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-qpf7p"]
Oct 30 18:47:37.974095 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:47:37.974066    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80115876_a6e1_4f39_a6c9_6847373e70f9.slice/crio-8c612b4526aa18993e70f6c99f5c49a931390e4593b3debe9b3156c6850a7b90 WatchSource:0}: Error finding container 8c612b4526aa18993e70f6c99f5c49a931390e4593b3debe9b3156c6850a7b90: Status 404 returned error can't find the container with id 8c612b4526aa18993e70f6c99f5c49a931390e4593b3debe9b3156c6850a7b90
Oct 30 18:47:37.975377 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:37.975356    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:47:38.196896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.196857    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"
Oct 30 18:47:38.196896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.196882    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2" event={"ID":"c49eba81-91bb-4472-af81-fef0cf06604e","Type":"ContainerDied","Data":"270fbde0519e6587c53e75fe52e4226e072551cd3763c00ea85b66e7b6f6072c"}
Oct 30 18:47:38.197467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.196947    4942 scope.go:122] "RemoveContainer" containerID="5e6399b8ad7e94b707d12780a5165615aa5bfba76706199a6f13a2afaf455412"
Oct 30 18:47:38.198309 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.198244    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p" event={"ID":"80115876-a6e1-4f39-a6c9-6847373e70f9","Type":"ContainerStarted","Data":"8c612b4526aa18993e70f6c99f5c49a931390e4593b3debe9b3156c6850a7b90"}
Oct 30 18:47:38.217862 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.217822    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:38.221804 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:38.221735    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-kt2k2"]
Oct 30 18:47:39.205063 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:39.205023    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p" event={"ID":"80115876-a6e1-4f39-a6c9-6847373e70f9","Type":"ContainerStarted","Data":"95bb7c2b4641f5da1f11040da7efea6e99058bcaa15ce707a4e6df512e8d5de6"}
Oct 30 18:47:39.541213 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:39.541132    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49eba81-91bb-4472-af81-fef0cf06604e" path="/var/lib/kubelet/pods/c49eba81-91bb-4472-af81-fef0cf06604e/volumes"
Oct 30 18:47:51.639967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.639906    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-685c9f986c-qpf7p" podStartSLOduration=14.321499472 podStartE2EDuration="14.6398923s" totalImagesPullingTime="318.392828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:39.229617892 +0000 UTC m=+645.807713584" watchObservedRunningTime="2026-10-30 18:47:51.6398923 +0000 UTC m=+658.217988062"
Oct 30 18:47:51.641155 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.641125    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7j28b"]
Oct 30 18:47:51.645287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.645252    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.658576 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.658532    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 18:47:51.659197 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.659170    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7j28b"]
Oct 30 18:47:51.659374 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.659352    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Oct 30 18:47:51.790292 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.790234    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s676\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-kube-api-access-9s676\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.790292 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.790296    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.790576 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.790410    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2002c0d-d6bf-470e-befb-53a919063a18-data\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.891915 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.891808    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2002c0d-d6bf-470e-befb-53a919063a18-data\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.892148 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.891931    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s676\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-kube-api-access-9s676\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.892148 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.891966    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.892285 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.892262    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2002c0d-d6bf-470e-befb-53a919063a18-data\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.894820 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.894773    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.904859 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.904819    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s676\" (UniqueName: \"kubernetes.io/projected/e2002c0d-d6bf-470e-befb-53a919063a18-kube-api-access-9s676\") pod \"seaweedfs-tls-serving-5c74974785-7j28b\" (UID: \"e2002c0d-d6bf-470e-befb-53a919063a18\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:51.959915 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:51.959870    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b"
Oct 30 18:47:52.248248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:52.248208    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7j28b"]
Oct 30 18:47:52.252837 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:47:52.252802    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2002c0d_d6bf_470e_befb_53a919063a18.slice/crio-67a5f5305992f54ed297ff78447be6d741b07fd5146317d7de16e63c2721f716 WatchSource:0}: Error finding container 67a5f5305992f54ed297ff78447be6d741b07fd5146317d7de16e63c2721f716: Status 404 returned error can't find the container with id 67a5f5305992f54ed297ff78447be6d741b07fd5146317d7de16e63c2721f716
Oct 30 18:47:52.264950 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:52.264905    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b" event={"ID":"e2002c0d-d6bf-470e-befb-53a919063a18","Type":"ContainerStarted","Data":"67a5f5305992f54ed297ff78447be6d741b07fd5146317d7de16e63c2721f716"}
Oct 30 18:47:53.271430 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:47:53.271381    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b" event={"ID":"e2002c0d-d6bf-470e-befb-53a919063a18","Type":"ContainerStarted","Data":"35926361e5f1bd995cb222d9d02bf463d3841d990a8cacd33e38d02d93798bbd"}
Oct 30 18:47:56.732381 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:47:56.732339    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:48:23.609129 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.609073    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-7j28b" podStartSLOduration=32.27854957 podStartE2EDuration="32.60905952s" totalImagesPullingTime="330.50995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:53.296174309 +0000 UTC m=+659.874270002" watchObservedRunningTime="2026-10-30 18:48:23.60905952 +0000 UTC m=+690.187155211"
Oct 30 18:48:23.609571 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.609203    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:48:23.613658 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.613636    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.622458 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.622437    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 18:48:23.623400 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623377    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 18:48:23.623400 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623392    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 30 18:48:23.623546 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623424    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 18:48:23.623546 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623441    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 30 18:48:23.623546 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623457    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 18:48:23.623546 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.623475    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:48:23.629286 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.629263    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:48:23.681419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.681381    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.681611 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.681437    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.681611 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.681466    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4tb\" (UniqueName: \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.681611 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.681553    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.681611 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.681590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782189 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782151    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782189 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782190    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4tb\" (UniqueName: \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782431 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782212    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782431 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782228    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782431 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782268    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.782849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.782809    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.783044 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.783024    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.783123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.783076    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.784734 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.784718    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.796338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.796316    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4tb\" (UniqueName: \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\") pod \"isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:23.932718 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:23.932693    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:24.217132 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:24.217055    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:48:24.220002 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:48:24.219973    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51ec2de_09dc_4447_afe5_726a33018190.slice/crio-7721a716604d2c7427b6fbe88b588dd9ed4b2c4190a4349b7ea106e767e7afd2 WatchSource:0}: Error finding container 7721a716604d2c7427b6fbe88b588dd9ed4b2c4190a4349b7ea106e767e7afd2: Status 404 returned error can't find the container with id 7721a716604d2c7427b6fbe88b588dd9ed4b2c4190a4349b7ea106e767e7afd2
Oct 30 18:48:24.411087 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:24.411048    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerStarted","Data":"7721a716604d2c7427b6fbe88b588dd9ed4b2c4190a4349b7ea106e767e7afd2"}
Oct 30 18:48:26.728631 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:48:26.728586    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:48:28.430067 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:28.430028    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerStarted","Data":"3fb5f87d64152f094475c572572d42bbffc0cd29c8e5f52ea5243d19b468125e"}
Oct 30 18:48:32.456893 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:32.456843    4942 generic.go:356] "Generic (PLEG): container finished" podID="c51ec2de-09dc-4447-afe5-726a33018190" containerID="3fb5f87d64152f094475c572572d42bbffc0cd29c8e5f52ea5243d19b468125e" exitCode=0
Oct 30 18:48:32.457309 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:32.456920    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerDied","Data":"3fb5f87d64152f094475c572572d42bbffc0cd29c8e5f52ea5243d19b468125e"}
Oct 30 18:48:48.541102 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:48.541053    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerStarted","Data":"4b9ae3ad8b82b78ad6340909601b71ea1ebbf2af117bdc2113b89cbf0dd8344d"}
Oct 30 18:48:51.557388 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:51.557352    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerStarted","Data":"339e088ebd28b34417d15588651123dfb4b58dd9ecf387194c559919e3881d48"}
Oct 30 18:48:56.724081 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:48:56.724040    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:48:57.586165 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:57.586124    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerStarted","Data":"de663b754896b53be634f1e1050937983dc92560a6104b72166300db0b009bc0"}
Oct 30 18:48:57.586371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:57.586301    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:57.592206 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:57.592182    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:57.661491 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:57.661436    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podStartSLOduration=3.316748619 podStartE2EDuration="34.661422288s" totalImagesPullingTime="27.207314983s" totalInitContainerRuntime="4.137358686s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:23 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:57.632686371 +0000 UTC m=+724.210782075" watchObservedRunningTime="2026-10-30 18:48:57.661422288 +0000 UTC m=+724.239517979"
Oct 30 18:48:58.590681 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:58.590641    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:58.591196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:58.590743    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:48:58.591196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:58.591126    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:48:58.591685 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:58.591654    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:48:59.594834 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:59.594791    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:48:59.595343 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:48:59.595168    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:09.595329 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:09.595274    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:09.595756 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:09.595664    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:19.595377 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:19.595332    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:19.595878 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:19.595773    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:26.724548 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:49:26.724512    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:49:29.595634 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:29.595589    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:29.596075 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:29.596046    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:39.595248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:39.595151    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:39.595630 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:39.595571    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:49.594956 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:49.594907    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:49.595497 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:49.595468    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:49:56.724611 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:49:56.724568    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:49:59.595604 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:59.595549    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:49:59.596130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:49:59.596058    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:07.537478 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:07.537422    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:50:07.537977 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:07.537953    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:17.541597 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:17.541567    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:17.541993 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:17.541653    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:20.719745 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.719702    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:50:20.723442 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.723425    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.736187 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.736168    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 18:50:20.736388 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.736373    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:50:20.736452 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.736406    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-primary-15f7e6-kube-rbac-proxy-sar-config\""
Oct 30 18:50:20.736642 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.736627    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 18:50:20.739589 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.739570    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-primary-15f7e6-predictor-serving-cert\""
Oct 30 18:50:20.739685 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.739645    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 18:50:20.739751 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.739710    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 18:50:20.740704 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.740684    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:50:20.775709 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.775684    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.775815 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.775724    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.775815 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.775746    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcnz\" (UniqueName: \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.775904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.775882    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.775941 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.775916    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.876892 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.876855    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.876892 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.876902    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.876925    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcnz\" (UniqueName: \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.876958    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877196 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.876973    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.877492    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877720 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.877701    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.877781 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.877745    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.879341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.879320    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:20.892516 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:20.892489    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcnz\" (UniqueName: \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\") pod \"isvc-primary-15f7e6-predictor-7bcd5db97c-sr786\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") " pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:21.040655 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:21.040576    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:21.323887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:21.323823    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:50:21.326724 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:50:21.326694    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba49cfd0_6fd8_4ef2_9269_834d16f071e5.slice/crio-80647eac6ff15e68824cc727f6f23cfeca126ff50c5dcc3a6f77098e7227b9ac WatchSource:0}: Error finding container 80647eac6ff15e68824cc727f6f23cfeca126ff50c5dcc3a6f77098e7227b9ac: Status 404 returned error can't find the container with id 80647eac6ff15e68824cc727f6f23cfeca126ff50c5dcc3a6f77098e7227b9ac
Oct 30 18:50:21.952648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:21.952605    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerStarted","Data":"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"}
Oct 30 18:50:21.953062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:21.952677    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerStarted","Data":"80647eac6ff15e68824cc727f6f23cfeca126ff50c5dcc3a6f77098e7227b9ac"}
Oct 30 18:50:25.970684 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:25.970651    4942 generic.go:356] "Generic (PLEG): container finished" podID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerID="0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae" exitCode=0
Oct 30 18:50:25.971101 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:25.970688    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerDied","Data":"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"}
Oct 30 18:50:26.724319 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:50:26.724274    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:50:26.978042 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:26.977950    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerStarted","Data":"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"}
Oct 30 18:50:26.978042 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:26.978028    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerStarted","Data":"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"}
Oct 30 18:50:26.978469 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:26.978240    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:27.006360 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:27.006313    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podStartSLOduration=3.03291719 podStartE2EDuration="7.006301527s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.973384337s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:27.003976263 +0000 UTC m=+813.582071955" watchObservedRunningTime="2026-10-30 18:50:27.006301527 +0000 UTC m=+813.584397312"
Oct 30 18:50:27.982690 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:27.982657    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:27.984016 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:27.983971    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:50:28.837392 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.837363    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:50:28.837678 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.837644    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" containerID="cri-o://4b9ae3ad8b82b78ad6340909601b71ea1ebbf2af117bdc2113b89cbf0dd8344d" gracePeriod=30
Oct 30 18:50:28.837678 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.837654    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" containerID="cri-o://de663b754896b53be634f1e1050937983dc92560a6104b72166300db0b009bc0" gracePeriod=30
Oct 30 18:50:28.837880 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.837673    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" containerID="cri-o://339e088ebd28b34417d15588651123dfb4b58dd9ecf387194c559919e3881d48" gracePeriod=30
Oct 30 18:50:28.991806 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.991769    4942 generic.go:356] "Generic (PLEG): container finished" podID="c51ec2de-09dc-4447-afe5-726a33018190" containerID="339e088ebd28b34417d15588651123dfb4b58dd9ecf387194c559919e3881d48" exitCode=0
Oct 30 18:50:28.992191 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.991845    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerDied","Data":"339e088ebd28b34417d15588651123dfb4b58dd9ecf387194c559919e3881d48"}
Oct 30 18:50:28.992233 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:28.992204    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:50:32.586735 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:32.586692    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:33.996586 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:33.996554    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:50:33.997105 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:33.997076    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:50:34.015958 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:34.015922    4942 generic.go:356] "Generic (PLEG): container finished" podID="c51ec2de-09dc-4447-afe5-726a33018190" containerID="4b9ae3ad8b82b78ad6340909601b71ea1ebbf2af117bdc2113b89cbf0dd8344d" exitCode=0
Oct 30 18:50:34.016149 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:34.015998    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerDied","Data":"4b9ae3ad8b82b78ad6340909601b71ea1ebbf2af117bdc2113b89cbf0dd8344d"}
Oct 30 18:50:37.537837 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:37.537797    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:50:37.538315 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:37.538178    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:37.586911 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:37.586875    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:42.587392 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:42.587346    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:42.587830 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:42.587515    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:43.997229 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:43.997193    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:50:47.537626 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:47.537586    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:50:47.538113 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:47.538028    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:47.587265 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:47.587231    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:52.587366 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:52.587310    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:53.997176 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:53.997136    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:50:56.724698 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:50:56.724659    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:50:57.537292 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:57.537251    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 30 18:50:57.537633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:57.537609    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:57.541900 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:57.541881    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:57.541970 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:57.541960    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:57.586510 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:57.586478    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 18:50:59.122416 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.122342    4942 generic.go:356] "Generic (PLEG): container finished" podID="c51ec2de-09dc-4447-afe5-726a33018190" containerID="de663b754896b53be634f1e1050937983dc92560a6104b72166300db0b009bc0" exitCode=0
Oct 30 18:50:59.122781 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.122418    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerDied","Data":"de663b754896b53be634f1e1050937983dc92560a6104b72166300db0b009bc0"}
Oct 30 18:50:59.479097 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.479072    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:50:59.595569 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595543    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\") pod \"c51ec2de-09dc-4447-afe5-726a33018190\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") "
Oct 30 18:50:59.595711 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595582    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\") pod \"c51ec2de-09dc-4447-afe5-726a33018190\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") "
Oct 30 18:50:59.595711 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595614    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\" (UniqueName: \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\") pod \"c51ec2de-09dc-4447-afe5-726a33018190\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") "
Oct 30 18:50:59.595711 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595644    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\") pod \"c51ec2de-09dc-4447-afe5-726a33018190\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") "
Oct 30 18:50:59.595711 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595685    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\") pod \"c51ec2de-09dc-4447-afe5-726a33018190\" (UID: \"c51ec2de-09dc-4447-afe5-726a33018190\") "
Oct 30 18:50:59.595920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595840    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location" pod "c51ec2de-09dc-4447-afe5-726a33018190" (UID: "c51ec2de-09dc-4447-afe5-726a33018190"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:59.595978 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.595945    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert" pod "c51ec2de-09dc-4447-afe5-726a33018190" (UID: "c51ec2de-09dc-4447-afe5-726a33018190"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:59.596159 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.596134    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config" pod "c51ec2de-09dc-4447-afe5-726a33018190" (UID: "c51ec2de-09dc-4447-afe5-726a33018190"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:59.597786 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.597763    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls" pod "c51ec2de-09dc-4447-afe5-726a33018190" (UID: "c51ec2de-09dc-4447-afe5-726a33018190"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:59.597864 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.597843    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb" pod "c51ec2de-09dc-4447-afe5-726a33018190" (UID: "c51ec2de-09dc-4447-afe5-726a33018190"). InnerVolumeSpecName "kube-api-access-hz4tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:59.696673 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.696645    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ec2de-09dc-4447-afe5-726a33018190-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:59.696673 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.696673    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:59.696832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.696683    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4tb\" (UniqueName: \"kubernetes.io/projected/c51ec2de-09dc-4447-afe5-726a33018190-kube-api-access-hz4tb\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:59.696832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.696695    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ec2de-09dc-4447-afe5-726a33018190-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:59.696832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:50:59.696703    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2de-09dc-4447-afe5-726a33018190-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:00.129703 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.129625    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk" event={"ID":"c51ec2de-09dc-4447-afe5-726a33018190","Type":"ContainerDied","Data":"7721a716604d2c7427b6fbe88b588dd9ed4b2c4190a4349b7ea106e767e7afd2"}
Oct 30 18:51:00.129703 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.129667    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"
Oct 30 18:51:00.129703 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.129697    4942 scope.go:122] "RemoveContainer" containerID="de663b754896b53be634f1e1050937983dc92560a6104b72166300db0b009bc0"
Oct 30 18:51:00.137980 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.137960    4942 scope.go:122] "RemoveContainer" containerID="339e088ebd28b34417d15588651123dfb4b58dd9ecf387194c559919e3881d48"
Oct 30 18:51:00.145556 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.145535    4942 scope.go:122] "RemoveContainer" containerID="4b9ae3ad8b82b78ad6340909601b71ea1ebbf2af117bdc2113b89cbf0dd8344d"
Oct 30 18:51:00.153131 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.153108    4942 scope.go:122] "RemoveContainer" containerID="3fb5f87d64152f094475c572572d42bbffc0cd29c8e5f52ea5243d19b468125e"
Oct 30 18:51:00.154780 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.154756    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:51:00.160341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.160318    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-predictor-8ff5b45bc-d48jk"]
Oct 30 18:51:00.672368 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672339    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:51:00.672696 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672683    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy"
Oct 30 18:51:00.672739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672697    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy"
Oct 30 18:51:00.672739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672726    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent"
Oct 30 18:51:00.672739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672732    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672740    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672746    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672754    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="storage-initializer"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672759    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="storage-initializer"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672820    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kube-rbac-proxy"
Oct 30 18:51:00.672832 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672829    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="agent"
Oct 30 18:51:00.673003 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.672836    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ec2de-09dc-4447-afe5-726a33018190" containerName="kserve-container"
Oct 30 18:51:00.677800 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.677782    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.684125 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684099    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 18:51:00.684240 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684206    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 30 18:51:00.684315 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684269    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:51:00.684372 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684350    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 18:51:00.684868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684752    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 30 18:51:00.684868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684801    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 18:51:00.684868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.684832    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 18:51:00.690927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.690904    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:51:00.705548 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.705527    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.705637 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.705562    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2g9\" (UniqueName: \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.705637 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.705581    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.705737 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.705718    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.705775 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.705766    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.806493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.806462    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.806679 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.806517    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.806679 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.806563    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.806679 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.806604    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2g9\" (UniqueName: \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.806854 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.806725    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.807120 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.807090    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.807224 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.807174    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.807315 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.807294    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.808916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.808898    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.823174 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.823156    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2g9\" (UniqueName: \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\") pod \"isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") " pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:00.998280 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:00.998256    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:01.281929 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:01.281779    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:51:01.284379 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:51:01.284351    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12606eb1_47ff_4edb_aff2_dd1cdf1680c0.slice/crio-65c0e7fb0273c5ab09b980285513de1ac816e9911a3e68454641e02f564a0f60 WatchSource:0}: Error finding container 65c0e7fb0273c5ab09b980285513de1ac816e9911a3e68454641e02f564a0f60: Status 404 returned error can't find the container with id 65c0e7fb0273c5ab09b980285513de1ac816e9911a3e68454641e02f564a0f60
Oct 30 18:51:01.542650 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:01.542572    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51ec2de-09dc-4447-afe5-726a33018190" path="/var/lib/kubelet/pods/c51ec2de-09dc-4447-afe5-726a33018190/volumes"
Oct 30 18:51:02.140853 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:02.140813    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerStarted","Data":"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"}
Oct 30 18:51:02.141064 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:02.140881    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerStarted","Data":"65c0e7fb0273c5ab09b980285513de1ac816e9911a3e68454641e02f564a0f60"}
Oct 30 18:51:03.997716 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:03.997630    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:51:05.156573 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:05.156540    4942 generic.go:356] "Generic (PLEG): container finished" podID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerID="bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199" exitCode=0
Oct 30 18:51:05.156897 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:05.156574    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerDied","Data":"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"}
Oct 30 18:51:06.164226 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.164192    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerStarted","Data":"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"}
Oct 30 18:51:06.164581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.164256    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerStarted","Data":"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"}
Oct 30 18:51:06.164581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.164281    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerStarted","Data":"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"}
Oct 30 18:51:06.164581 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.164554    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:06.164693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.164602    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:06.165907 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:06.165885    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:07.169063 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:07.169025    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:07.169467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:07.169168    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:07.170264 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:07.170233    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:08.173607 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:08.173564    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:08.174002 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:08.173945    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:13.177670 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:13.177641    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:51:13.178280 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:13.178249    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:13.178569 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:13.178546    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:13.206569 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:13.206527    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podStartSLOduration=9.459528213 podStartE2EDuration="13.206516621s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746988408s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:06.202095937 +0000 UTC m=+852.780191631" watchObservedRunningTime="2026-10-30 18:51:13.206516621 +0000 UTC m=+859.784612375"
Oct 30 18:51:13.997174 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:13.997135    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:51:23.178496 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:23.178452    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:23.178981 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:23.178946    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:23.997682 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:23.997647    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:51:26.724948 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:51:26.724905    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:51:33.178231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:33.178187    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:33.180551 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:33.178645    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:33.997226 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:33.997189    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Oct 30 18:51:43.178412 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:43.178369    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:43.178822 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:43.178798    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:43.998053 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:43.998003    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:51:53.178650 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:53.178612    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:51:53.179136 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:53.179099    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:53.610453 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:53.610366    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:51:53.612842 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:51:53.612814    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:51:56.724502 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:51:56.724464    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:52:03.178294 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:03.178245    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:03.178809 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:03.178784    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:06.191218 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.191176    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:06.196015 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.195987    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.203186 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.203161    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-secondary-15f7e6-kube-rbac-proxy-sar-config\""
Oct 30 18:52:06.204553 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.204528    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-secondary-15f7e6-predictor-serving-cert\""
Oct 30 18:52:06.204649 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.204579    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"invalid-s3-secret-15f7e6\""
Oct 30 18:52:06.204649 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.204583    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"invalid-s3-sa-15f7e6-dockercfg-p5zrk\""
Oct 30 18:52:06.209412 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.209388    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:06.241126 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.241090    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.241126 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.241128    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.241342 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.241153    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.241342 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.241275    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2mj\" (UniqueName: \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.241342 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.241304    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342646 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.342599    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342646 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.342640    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.342669    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.342703    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2mj\" (UniqueName: \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.342718    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.342920 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:06.342778    4942 secret.go:189] Couldn't get secret e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-serving-cert: secret "isvc-secondary-15f7e6-predictor-serving-cert" not found
Oct 30 18:52:06.342920 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:06.342885    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls podName:be351769-1317-457c-9d83-165ac38c793c nodeName:}" failed. No retries permitted until 2026-10-30 18:52:06.842863698 +0000 UTC m=+913.420959369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls") pod "isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" (UID: "be351769-1317-457c-9d83-165ac38c793c") : secret "isvc-secondary-15f7e6-predictor-serving-cert" not found
Oct 30 18:52:06.343238 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.343219    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.343420 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.343402    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.343502 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.343479    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.356275 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.356242    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2mj\" (UniqueName: \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.847737 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.847690    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:06.850317 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:06.850283    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") pod \"isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") " pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:07.113675 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:07.113573    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:07.404886 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:07.404857    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:07.407192 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:52:07.407150    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe351769_1317_457c_9d83_165ac38c793c.slice/crio-fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2 WatchSource:0}: Error finding container fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2: Status 404 returned error can't find the container with id fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2
Oct 30 18:52:07.423825 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:07.423784    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerStarted","Data":"fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2"}
Oct 30 18:52:08.429423 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:08.429383    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerStarted","Data":"14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"}
Oct 30 18:52:13.178459 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:13.178410    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:13.178929 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:13.178905    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:14.459226 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:14.459192    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/0.log"
Oct 30 18:52:14.459663 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:14.459237    4942 generic.go:356] "Generic (PLEG): container finished" podID="be351769-1317-457c-9d83-165ac38c793c" containerID="14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3" exitCode=1
Oct 30 18:52:14.459663 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:14.459316    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerDied","Data":"14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"}
Oct 30 18:52:15.466073 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:15.466045    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/0.log"
Oct 30 18:52:15.466464 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:15.466126    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerStarted","Data":"24110b1a0328bb435b749d15d478813551120fa7dd7e77d83a7bf2c07288387f"}
Oct 30 18:52:17.536998 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:17.536953    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:17.537472 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:17.537378    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:19.484258 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.484226    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/1.log"
Oct 30 18:52:19.484695 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.484676    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/0.log"
Oct 30 18:52:19.484737 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.484714    4942 generic.go:356] "Generic (PLEG): container finished" podID="be351769-1317-457c-9d83-165ac38c793c" containerID="24110b1a0328bb435b749d15d478813551120fa7dd7e77d83a7bf2c07288387f" exitCode=1
Oct 30 18:52:19.484784 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.484770    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerDied","Data":"24110b1a0328bb435b749d15d478813551120fa7dd7e77d83a7bf2c07288387f"}
Oct 30 18:52:19.484839 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.484829    4942 scope.go:122] "RemoveContainer" containerID="14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"
Oct 30 18:52:19.485245 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.485218    4942 scope.go:122] "RemoveContainer" containerID="14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"
Oct 30 18:52:19.496434 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:19.496393    4942 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_e2e-gw1-c83bbf5b66(be351769-1317-457c-9d83-165ac38c793c)\"" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" podUID="be351769-1317-457c-9d83-165ac38c793c"
Oct 30 18:52:19.496949 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:19.496919    4942 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_e2e-gw1-c83bbf5b66_be351769-1317-457c-9d83-165ac38c793c_0 in pod sandbox fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2 from index: no such id: '14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3'" containerID="14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"
Oct 30 18:52:19.497075 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:19.496962    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_e2e-gw1-c83bbf5b66_be351769-1317-457c-9d83-165ac38c793c_0 in pod sandbox fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2 from index: no such id: '14332529d6b6853acd0847e17dd97e9b131ccb6ad239a2599e92033661e1a8b3'"
Oct 30 18:52:20.491446 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:20.491412    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/1.log"
Oct 30 18:52:26.328463 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.328420    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:26.379778 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.379734    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:52:26.380090 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.380037    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container" containerID="cri-o://23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b" gracePeriod=30
Oct 30 18:52:26.380245 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.380197    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kube-rbac-proxy" containerID="cri-o://296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4" gracePeriod=30
Oct 30 18:52:26.505607 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.505573    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/1.log"
Oct 30 18:52:26.507941 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.507906    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:26.522482 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.522448    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl_be351769-1317-457c-9d83-165ac38c793c/storage-initializer/1.log"
Oct 30 18:52:26.522698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.522561    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"
Oct 30 18:52:26.522698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.522571    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl" event={"ID":"be351769-1317-457c-9d83-165ac38c793c","Type":"ContainerDied","Data":"fdafc492029df99ead7d57976fb69c2b25d4b8c7fb572b50e01e8a81c6ee76d2"}
Oct 30 18:52:26.522698 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.522648    4942 scope.go:122] "RemoveContainer" containerID="24110b1a0328bb435b749d15d478813551120fa7dd7e77d83a7bf2c07288387f"
Oct 30 18:52:26.525324 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.525292    4942 generic.go:356] "Generic (PLEG): container finished" podID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerID="296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4" exitCode=0
Oct 30 18:52:26.525482 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.525337    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerDied","Data":"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"}
Oct 30 18:52:26.625745 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625631    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\") pod \"be351769-1317-457c-9d83-165ac38c793c\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") "
Oct 30 18:52:26.625908 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625776    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\" (UniqueName: \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\") pod \"be351769-1317-457c-9d83-165ac38c793c\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") "
Oct 30 18:52:26.625908 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625826    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") pod \"be351769-1317-457c-9d83-165ac38c793c\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") "
Oct 30 18:52:26.625908 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625877    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\") pod \"be351769-1317-457c-9d83-165ac38c793c\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") "
Oct 30 18:52:26.626139 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625911    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\") pod \"be351769-1317-457c-9d83-165ac38c793c\" (UID: \"be351769-1317-457c-9d83-165ac38c793c\") "
Oct 30 18:52:26.626139 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.625944    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location" pod "be351769-1317-457c-9d83-165ac38c793c" (UID: "be351769-1317-457c-9d83-165ac38c793c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:26.626247 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.626218    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be351769-1317-457c-9d83-165ac38c793c-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:26.626307 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.626269    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config" pod "be351769-1317-457c-9d83-165ac38c793c" (UID: "be351769-1317-457c-9d83-165ac38c793c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:26.626355 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.626324    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert" pod "be351769-1317-457c-9d83-165ac38c793c" (UID: "be351769-1317-457c-9d83-165ac38c793c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:26.628436 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.628392    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls" pod "be351769-1317-457c-9d83-165ac38c793c" (UID: "be351769-1317-457c-9d83-165ac38c793c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:26.628548 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.628397    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj" pod "be351769-1317-457c-9d83-165ac38c793c" (UID: "be351769-1317-457c-9d83-165ac38c793c"). InnerVolumeSpecName "kube-api-access-qp2mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:26.726647 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.726608    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be351769-1317-457c-9d83-165ac38c793c-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:26.726647 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.726644    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:26.726863 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.726660    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be351769-1317-457c-9d83-165ac38c793c-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:26.726863 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.726673    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2mj\" (UniqueName: \"kubernetes.io/projected/be351769-1317-457c-9d83-165ac38c793c-kube-api-access-qp2mj\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:26.728494 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:26.728453    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:52:26.873496 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.873442    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:26.878259 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:26.878152    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-secondary-15f7e6-predictor-cd865dd7d-xh2pl"]
Oct 30 18:52:27.542904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:27.542866    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be351769-1317-457c-9d83-165ac38c793c" path="/var/lib/kubelet/pods/be351769-1317-457c-9d83-165ac38c793c/volumes"
Oct 30 18:52:27.543352 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:27.543265    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:52:27.543396 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:27.543362    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:52:28.992820 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:28.992765    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 30 18:52:32.335980 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.335941    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:52:32.478565 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478522    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\") pod \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") "
Oct 30 18:52:32.478760 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478578    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\") pod \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") "
Oct 30 18:52:32.478760 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478610    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\" (UniqueName: \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\") pod \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") "
Oct 30 18:52:32.478760 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478630    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\") pod \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") "
Oct 30 18:52:32.478760 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478659    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\") pod \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\" (UID: \"ba49cfd0-6fd8-4ef2-9269-834d16f071e5\") "
Oct 30 18:52:32.479103 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.478988    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config" pod "ba49cfd0-6fd8-4ef2-9269-834d16f071e5" (UID: "ba49cfd0-6fd8-4ef2-9269-834d16f071e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:32.479248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.479083    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert" pod "ba49cfd0-6fd8-4ef2-9269-834d16f071e5" (UID: "ba49cfd0-6fd8-4ef2-9269-834d16f071e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:32.479248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.479122    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location" pod "ba49cfd0-6fd8-4ef2-9269-834d16f071e5" (UID: "ba49cfd0-6fd8-4ef2-9269-834d16f071e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:32.481054 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.481020    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls" pod "ba49cfd0-6fd8-4ef2-9269-834d16f071e5" (UID: "ba49cfd0-6fd8-4ef2-9269-834d16f071e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:32.481195 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.481049    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz" pod "ba49cfd0-6fd8-4ef2-9269-834d16f071e5" (UID: "ba49cfd0-6fd8-4ef2-9269-834d16f071e5"). InnerVolumeSpecName "kube-api-access-5vcnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:32.559896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.559857    4942 generic.go:356] "Generic (PLEG): container finished" podID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerID="23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b" exitCode=0
Oct 30 18:52:32.559896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.559902    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerDied","Data":"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"}
Oct 30 18:52:32.560204 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.559954    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786" event={"ID":"ba49cfd0-6fd8-4ef2-9269-834d16f071e5","Type":"ContainerDied","Data":"80647eac6ff15e68824cc727f6f23cfeca126ff50c5dcc3a6f77098e7227b9ac"}
Oct 30 18:52:32.560204 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.559982    4942 scope.go:122] "RemoveContainer" containerID="296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"
Oct 30 18:52:32.560204 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.559992    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"
Oct 30 18:52:32.569927 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.569902    4942 scope.go:122] "RemoveContainer" containerID="23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"
Oct 30 18:52:32.579149 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579121    4942 scope.go:122] "RemoveContainer" containerID="0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"
Oct 30 18:52:32.579500 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579475    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:32.579562 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579505    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:32.579562 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579521    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcnz\" (UniqueName: \"kubernetes.io/projected/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kube-api-access-5vcnz\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:32.579562 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579530    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:32.579562 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.579539    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba49cfd0-6fd8-4ef2-9269-834d16f071e5-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:32.587532 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.587490    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:52:32.588747 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.588727    4942 scope.go:122] "RemoveContainer" containerID="296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"
Oct 30 18:52:32.589107 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:32.589087    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4\": container with ID starting with 296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4 not found: ID does not exist" containerID="296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"
Oct 30 18:52:32.589159 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.589121    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4"} err="failed to get container status \"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4\": rpc error: code = NotFound desc = could not find container \"296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4\": container with ID starting with 296c08861f7c7c8f2398f3d92fbd9f1a3c9f82cd60809c82b9d12047809f2ec4 not found: ID does not exist"
Oct 30 18:52:32.589159 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.589140    4942 scope.go:122] "RemoveContainer" containerID="23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"
Oct 30 18:52:32.589407 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:32.589385    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b\": container with ID starting with 23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b not found: ID does not exist" containerID="23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"
Oct 30 18:52:32.589462 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.589415    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b"} err="failed to get container status \"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b\": rpc error: code = NotFound desc = could not find container \"23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b\": container with ID starting with 23f44c643e78413a736fcc45389d7dff8a05db6fa14e11ea4b41493b99633b7b not found: ID does not exist"
Oct 30 18:52:32.589462 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.589436    4942 scope.go:122] "RemoveContainer" containerID="0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"
Oct 30 18:52:32.589749 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:32.589731    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae\": container with ID starting with 0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae not found: ID does not exist" containerID="0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"
Oct 30 18:52:32.589792 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.589754    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae"} err="failed to get container status \"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae\": rpc error: code = NotFound desc = could not find container \"0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae\": container with ID starting with 0611a4dd1d441836440e0290c46a5a4f2808e21e9c335bbd9af642117da10eae not found: ID does not exist"
Oct 30 18:52:32.594625 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:32.594587    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-primary-15f7e6-predictor-7bcd5db97c-sr786"]
Oct 30 18:52:33.543960 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:33.543900    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" path="/var/lib/kubelet/pods/ba49cfd0-6fd8-4ef2-9269-834d16f071e5/volumes"
Oct 30 18:52:34.119896 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.119857    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:34.120288 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120270    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kube-rbac-proxy"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120290    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kube-rbac-proxy"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120301    4942 cpu_manager.go:414] "containerMap: removing container" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120307    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120329    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120336    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120343    4942 cpu_manager.go:414] "containerMap: removing container" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120350    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120362    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container"
Oct 30 18:52:34.120378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120368    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container"
Oct 30 18:52:34.120652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120431    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kube-rbac-proxy"
Oct 30 18:52:34.120652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120441    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120447    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="be351769-1317-457c-9d83-165ac38c793c" containerName="storage-initializer"
Oct 30 18:52:34.120652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.120453    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49cfd0-6fd8-4ef2-9269-834d16f071e5" containerName="kserve-container"
Oct 30 18:52:34.125317 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.125287    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.139062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.138988    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:34.139214 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.138997    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-init-fail-ba7372-kube-rbac-proxy-sar-config\""
Oct 30 18:52:34.139214 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139002    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 18:52:34.139319 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139298    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:52:34.139365 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139323    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 18:52:34.139884 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139866    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-init-fail-ba7372-predictor-serving-cert\""
Oct 30 18:52:34.139937 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139898    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"fail-s3-secret-ba7372\""
Oct 30 18:52:34.139937 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.139915    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"fail-s3-sa-ba7372-dockercfg-jwrhj\""
Oct 30 18:52:34.294772 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.294715    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.294987 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.294793    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.294987 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.294886    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9kb\" (UniqueName: \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.294987 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.294941    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.295186 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.295058    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396134    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396195    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396227    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396522 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396247    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9kb\" (UniqueName: \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396522 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396276    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396707 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396681    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.396963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396940    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.397053 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.396949    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.399233 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.399206    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.410653 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.410619    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9kb\" (UniqueName: \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\") pod \"isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") " pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.444816 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.444772    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:34.734032 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:34.733983    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:34.737291 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:52:34.737245    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4cba4f_d885_4ab6_9f23_298f0733b1d0.slice/crio-880d7cbaa66c6709d8cb36c832f58b53b6581e2d6a6ae2d57b7448879a8fd3c3 WatchSource:0}: Error finding container 880d7cbaa66c6709d8cb36c832f58b53b6581e2d6a6ae2d57b7448879a8fd3c3: Status 404 returned error can't find the container with id 880d7cbaa66c6709d8cb36c832f58b53b6581e2d6a6ae2d57b7448879a8fd3c3
Oct 30 18:52:35.579945 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:35.579902    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerStarted","Data":"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"}
Oct 30 18:52:35.580193 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:35.579965    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerStarted","Data":"880d7cbaa66c6709d8cb36c832f58b53b6581e2d6a6ae2d57b7448879a8fd3c3"}
Oct 30 18:52:36.541996 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:36.541947    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:52:36.542453 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:36.542332    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" containerID="cri-o://cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb" gracePeriod=30
Oct 30 18:52:36.542453 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:36.542407    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" containerID="cri-o://90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305" gracePeriod=30
Oct 30 18:52:36.542453 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:36.542409    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" containerID="cri-o://5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1" gracePeriod=30
Oct 30 18:52:37.536987 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:37.536932    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:37.537419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:37.537392    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:37.593376 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:37.593336    4942 generic.go:356] "Generic (PLEG): container finished" podID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerID="5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1" exitCode=0
Oct 30 18:52:37.593821 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:37.593412    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerDied","Data":"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"}
Oct 30 18:52:38.173868 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:38.173812    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:52:40.612889 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:40.612837    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/0.log"
Oct 30 18:52:40.613351 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:40.612900    4942 generic.go:356] "Generic (PLEG): container finished" podID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerID="638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb" exitCode=1
Oct 30 18:52:40.613351 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:40.612973    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerDied","Data":"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"}
Oct 30 18:52:41.619841 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:41.619803    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/0.log"
Oct 30 18:52:41.620390 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:41.619914    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerStarted","Data":"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"}
Oct 30 18:52:43.174276 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:43.174225    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:52:43.632579 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:43.632538    4942 generic.go:356] "Generic (PLEG): container finished" podID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerID="cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb" exitCode=0
Oct 30 18:52:43.632767 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:43.632612    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerDied","Data":"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"}
Oct 30 18:52:44.162248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.162208    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:44.162502 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.162458    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer" containerID="cri-o://1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8" gracePeriod=30
Oct 30 18:52:44.327476 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.327443    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/1.log"
Oct 30 18:52:44.327935 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.327915    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/0.log"
Oct 30 18:52:44.330360 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.330333    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:44.384812 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.384765    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\") pod \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") "
Oct 30 18:52:44.384812 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.384813    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\") pod \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") "
Oct 30 18:52:44.385128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.384838    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\") pod \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") "
Oct 30 18:52:44.385128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.384960    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\") pod \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") "
Oct 30 18:52:44.385128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385093    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\" (UniqueName: \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\") pod \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\" (UID: \"fb4cba4f-d885-4ab6-9f23-298f0733b1d0\") "
Oct 30 18:52:44.385310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385259    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location" pod "fb4cba4f-d885-4ab6-9f23-298f0733b1d0" (UID: "fb4cba4f-d885-4ab6-9f23-298f0733b1d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:44.385310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385285    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert" pod "fb4cba4f-d885-4ab6-9f23-298f0733b1d0" (UID: "fb4cba4f-d885-4ab6-9f23-298f0733b1d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:44.385310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385266    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config" pod "fb4cba4f-d885-4ab6-9f23-298f0733b1d0" (UID: "fb4cba4f-d885-4ab6-9f23-298f0733b1d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:44.385431 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385401    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:44.385431 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385421    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:44.385504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.385435    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:44.387408 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.387371    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb" pod "fb4cba4f-d885-4ab6-9f23-298f0733b1d0" (UID: "fb4cba4f-d885-4ab6-9f23-298f0733b1d0"). InnerVolumeSpecName "kube-api-access-wq9kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:44.387528 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.387387    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls" pod "fb4cba4f-d885-4ab6-9f23-298f0733b1d0" (UID: "fb4cba4f-d885-4ab6-9f23-298f0733b1d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:44.486202 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.486163    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9kb\" (UniqueName: \"kubernetes.io/projected/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-kube-api-access-wq9kb\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:44.486202 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.486200    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cba4f-d885-4ab6-9f23-298f0733b1d0-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:44.639397 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639353    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/1.log"
Oct 30 18:52:44.639815 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639796    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45_fb4cba4f-d885-4ab6-9f23-298f0733b1d0/storage-initializer/0.log"
Oct 30 18:52:44.639870 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639835    4942 generic.go:356] "Generic (PLEG): container finished" podID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerID="1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8" exitCode=1
Oct 30 18:52:44.639906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639875    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerDied","Data":"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"}
Oct 30 18:52:44.639906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639900    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"
Oct 30 18:52:44.639971 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639945    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45" event={"ID":"fb4cba4f-d885-4ab6-9f23-298f0733b1d0","Type":"ContainerDied","Data":"880d7cbaa66c6709d8cb36c832f58b53b6581e2d6a6ae2d57b7448879a8fd3c3"}
Oct 30 18:52:44.640038 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.639977    4942 scope.go:122] "RemoveContainer" containerID="1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"
Oct 30 18:52:44.651118 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.651094    4942 scope.go:122] "RemoveContainer" containerID="638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"
Oct 30 18:52:44.660673 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.660642    4942 scope.go:122] "RemoveContainer" containerID="1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"
Oct 30 18:52:44.661054 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:44.660981    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8\": container with ID starting with 1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8 not found: ID does not exist" containerID="1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"
Oct 30 18:52:44.661129 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.661106    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8"} err="failed to get container status \"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8\": rpc error: code = NotFound desc = could not find container \"1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8\": container with ID starting with 1842c6742e82942d1722358e8796dcd1b3fe0356d9a51b09a6a424da1171b9d8 not found: ID does not exist"
Oct 30 18:52:44.661181 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.661131    4942 scope.go:122] "RemoveContainer" containerID="638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"
Oct 30 18:52:44.661483 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:44.661460    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb\": container with ID starting with 638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb not found: ID does not exist" containerID="638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"
Oct 30 18:52:44.661567 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.661488    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb"} err="failed to get container status \"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb\": rpc error: code = NotFound desc = could not find container \"638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb\": container with ID starting with 638c7a5739f1b5ab34950924f5efd9d59b6f8b4d1dedda9764871ff21c2eeffb not found: ID does not exist"
Oct 30 18:52:44.687217 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.687177    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:44.691319 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:44.691268    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-init-fail-ba7372-predictor-54fc6b4c76-vkw45"]
Oct 30 18:52:45.547059 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:45.545852    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" path="/var/lib/kubelet/pods/fb4cba4f-d885-4ab6-9f23-298f0733b1d0/volumes"
Oct 30 18:52:47.537172 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:47.537125    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:47.537645 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:47.537591    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:48.174605 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:48.174552    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:52:48.174808 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:48.174742    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:52:53.174368 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:53.174312    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:52:56.728770 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:52:56.728724    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:52:57.537028 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:57.536967    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:5000: connect: connection refused"
Oct 30 18:52:57.537419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:57.537392    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:57.541493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:57.541466    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:52:57.541637 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:57.541578    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:52:58.173973 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:52:58.173921    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:53:03.174602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:03.174557    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 18:53:06.719032 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.718978    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:53:06.743120 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.743084    4942 generic.go:356] "Generic (PLEG): container finished" podID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerID="90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305" exitCode=0
Oct 30 18:53:06.743287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.743157    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerDied","Data":"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"}
Oct 30 18:53:06.743287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.743180    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"
Oct 30 18:53:06.743287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.743231    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg" event={"ID":"12606eb1-47ff-4edb-aff2-dd1cdf1680c0","Type":"ContainerDied","Data":"65c0e7fb0273c5ab09b980285513de1ac816e9911a3e68454641e02f564a0f60"}
Oct 30 18:53:06.743287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.743249    4942 scope.go:122] "RemoveContainer" containerID="90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"
Oct 30 18:53:06.753360 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.753335    4942 scope.go:122] "RemoveContainer" containerID="5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"
Oct 30 18:53:06.764866 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.764825    4942 scope.go:122] "RemoveContainer" containerID="cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"
Oct 30 18:53:06.777425 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.777386    4942 scope.go:122] "RemoveContainer" containerID="bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"
Oct 30 18:53:06.784461 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784425    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\") pod \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") "
Oct 30 18:53:06.784593 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784534    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\") pod \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") "
Oct 30 18:53:06.784593 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784565    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\") pod \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") "
Oct 30 18:53:06.784664 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784621    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\" (UniqueName: \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\") pod \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") "
Oct 30 18:53:06.784707 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784655    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\") pod \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\" (UID: \"12606eb1-47ff-4edb-aff2-dd1cdf1680c0\") "
Oct 30 18:53:06.784997 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784939    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert" pod "12606eb1-47ff-4edb-aff2-dd1cdf1680c0" (UID: "12606eb1-47ff-4edb-aff2-dd1cdf1680c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:06.784997 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.784980    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location" pod "12606eb1-47ff-4edb-aff2-dd1cdf1680c0" (UID: "12606eb1-47ff-4edb-aff2-dd1cdf1680c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:06.785269 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.785235    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config" pod "12606eb1-47ff-4edb-aff2-dd1cdf1680c0" (UID: "12606eb1-47ff-4edb-aff2-dd1cdf1680c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:06.786904 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.786867    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls" pod "12606eb1-47ff-4edb-aff2-dd1cdf1680c0" (UID: "12606eb1-47ff-4edb-aff2-dd1cdf1680c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:06.787157 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.787137    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9" pod "12606eb1-47ff-4edb-aff2-dd1cdf1680c0" (UID: "12606eb1-47ff-4edb-aff2-dd1cdf1680c0"). InnerVolumeSpecName "kube-api-access-xz2g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:06.788191 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.788163    4942 scope.go:122] "RemoveContainer" containerID="90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"
Oct 30 18:53:06.788571 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:06.788550    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305\": container with ID starting with 90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305 not found: ID does not exist" containerID="90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"
Oct 30 18:53:06.788669 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.788586    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305"} err="failed to get container status \"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305\": rpc error: code = NotFound desc = could not find container \"90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305\": container with ID starting with 90d82de0f095e96c763936ffba0c4187c4b167e06b1c248ea9e0a002243f8305 not found: ID does not exist"
Oct 30 18:53:06.788669 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.788613    4942 scope.go:122] "RemoveContainer" containerID="5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"
Oct 30 18:53:06.788925 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:06.788905    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1\": container with ID starting with 5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1 not found: ID does not exist" containerID="5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"
Oct 30 18:53:06.788968 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.788932    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1"} err="failed to get container status \"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1\": rpc error: code = NotFound desc = could not find container \"5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1\": container with ID starting with 5a5e41a5836286db5e5dcd2013633df9e02327c8bcc6aa9a8840f2b72ec054b1 not found: ID does not exist"
Oct 30 18:53:06.788968 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.788949    4942 scope.go:122] "RemoveContainer" containerID="cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"
Oct 30 18:53:06.789322 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:06.789300    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb\": container with ID starting with cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb not found: ID does not exist" containerID="cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"
Oct 30 18:53:06.789389 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.789324    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb"} err="failed to get container status \"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb\": rpc error: code = NotFound desc = could not find container \"cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb\": container with ID starting with cb32c13fd5e3065ce9d4beeb533c07d0fab7b49d79a9b243477a36c61a3dd1fb not found: ID does not exist"
Oct 30 18:53:06.789389 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.789339    4942 scope.go:122] "RemoveContainer" containerID="bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"
Oct 30 18:53:06.789618 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:06.789594    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199\": container with ID starting with bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199 not found: ID does not exist" containerID="bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"
Oct 30 18:53:06.789678 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.789626    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199"} err="failed to get container status \"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199\": rpc error: code = NotFound desc = could not find container \"bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199\": container with ID starting with bcdb0fb083c944205bedc6259ceb7e4e65a638d706d8430605e3a6cbe41b0199 not found: ID does not exist"
Oct 30 18:53:06.885375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.885276    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:06.885375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.885310    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:06.885375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.885321    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2g9\" (UniqueName: \"kubernetes.io/projected/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-api-access-xz2g9\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:06.885375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.885335    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:06.885375 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:06.885348    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12606eb1-47ff-4edb-aff2-dd1cdf1680c0-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:07.069069 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:07.069003    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:53:07.072487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:07.072445    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-sklearn-batcher-custom-predictor-6b4bb5f64c-zsjxg"]
Oct 30 18:53:07.543655 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:07.543596    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" path="/var/lib/kubelet/pods/12606eb1-47ff-4edb-aff2-dd1cdf1680c0/volumes"
Oct 30 18:53:08.506937 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.506893    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507274    4942 cpu_manager.go:414] "containerMap: removing container" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507289    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507303    4942 cpu_manager.go:414] "containerMap: removing container" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507309    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507316    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507321    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507338    4942 cpu_manager.go:414] "containerMap: removing container" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507343    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507355    4942 cpu_manager.go:414] "containerMap: removing container" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="storage-initializer"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507361    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="storage-initializer"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507415    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="agent"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507425    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kube-rbac-proxy"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507431    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507438    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="12606eb1-47ff-4edb-aff2-dd1cdf1680c0" containerName="kserve-container"
Oct 30 18:53:08.507640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.507444    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:08.511887 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.511849    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:08.519672 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.519629    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 18:53:08.523806 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.523756    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:53:08.523973 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.523843    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 18:53:08.601151 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.601093    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdt4\" (UniqueName: \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\") pod \"proxy-routing-b24077bc-sink-885bf9646-sqxxq\" (UID: \"0443b9b1-b318-4f22-8597-2b4d8e564310\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:08.619965 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.619933    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 18:53:08.702600 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.702558    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdt4\" (UniqueName: \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\") pod \"proxy-routing-b24077bc-sink-885bf9646-sqxxq\" (UID: \"0443b9b1-b318-4f22-8597-2b4d8e564310\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:08.716736 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.716686    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdt4\" (UniqueName: \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\") pod \"proxy-routing-b24077bc-sink-885bf9646-sqxxq\" (UID: \"0443b9b1-b318-4f22-8597-2b4d8e564310\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:08.826963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:08.826857    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:09.113001 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:09.112902    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:53:09.116977 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:53:09.116936    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0443b9b1_b318_4f22_8597_2b4d8e564310.slice/crio-80890433c03fb88bc50b2ae083ccc86c7c080d395c6fb473dc5bb40c4014e90a WatchSource:0}: Error finding container 80890433c03fb88bc50b2ae083ccc86c7c080d395c6fb473dc5bb40c4014e90a: Status 404 returned error can't find the container with id 80890433c03fb88bc50b2ae083ccc86c7c080d395c6fb473dc5bb40c4014e90a
Oct 30 18:53:09.119523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:09.119497    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:53:09.763700 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:09.763655    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" event={"ID":"0443b9b1-b318-4f22-8597-2b4d8e564310","Type":"ContainerStarted","Data":"80890433c03fb88bc50b2ae083ccc86c7c080d395c6fb473dc5bb40c4014e90a"}
Oct 30 18:53:11.775666 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:11.775622    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" event={"ID":"0443b9b1-b318-4f22-8597-2b4d8e564310","Type":"ContainerStarted","Data":"d94b7c8faafdf63894db2ce38bf351f08ef8c4f90d56b76a5138acf6d44141d3"}
Oct 30 18:53:11.776117 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:11.775803    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:11.778539 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:11.778508    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:53:11.796637 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:11.796566    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" podStartSLOduration=1.587199402 podStartE2EDuration="3.796552327s" totalImagesPullingTime="2.209352925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:11.794516826 +0000 UTC m=+978.372612520" watchObservedRunningTime="2026-10-30 18:53:11.796552327 +0000 UTC m=+978.374648019"
Oct 30 18:53:15.999036 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:15.998982    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:53:15.999484 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:15.999398    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:15.999484 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:15.999412    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4cba4f-d885-4ab6-9f23-298f0733b1d0" containerName="storage-initializer"
Oct 30 18:53:16.002848 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.002825    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.009512 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.009474    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-b24077bc-dockercfg-jlmsb\""
Oct 30 18:53:16.010996 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.010976    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-b24077bc-predictor-serving-cert\""
Oct 30 18:53:16.011117 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.010994    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-b24077bc-kube-rbac-proxy-sar-config\""
Oct 30 18:53:16.016741 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.016706    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:53:16.169739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.169700    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttl65\" (UniqueName: \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.169932 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.169765    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.169932 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.169906    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.271227 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.271132    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.271227 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.271212    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.271429 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.271264    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttl65\" (UniqueName: \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.271876 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.271853    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.273831 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.273804    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:16.284925 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:16.284890    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttl65\" (UniqueName: \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.531130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.531082    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.541297 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.541267    4942 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") device path: \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.632561 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.632528    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.632728 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.632639    4942 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") DevicePath \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.636628 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.636605    4942 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") DevicePath \"csi-421b7f3edd787477bd15de95fc25eab8d331b23e04a7494920f903dc8339453a\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.812267 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.812230    4942 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/da258e1325708af2752266b545378d15f8443367f8fe75ee03c9f56de6b09326/globalmount\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:23.827522 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:23.827491    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-6bd4448555-ltp7n\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:24.119897 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.119780    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:24.403496 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.403425    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:53:24.406815 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:53:24.406769    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497 WatchSource:0}: Error finding container eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497: Status 404 returned error can't find the container with id eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497
Oct 30 18:53:24.832070 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.832028    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerStarted","Data":"9b7b725e91be2eb57dc551f3dfa5fea469f50107f1ed29519bce552e1f40e779"}
Oct 30 18:53:24.832487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.832099    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerStarted","Data":"30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a"}
Oct 30 18:53:24.832487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.832126    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerStarted","Data":"eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497"}
Oct 30 18:53:24.832487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.832157    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:24.861140 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:24.861091    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podStartSLOduration=12.861077152 podStartE2EDuration="12.861077152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:24.858759768 +0000 UTC m=+991.436855460" watchObservedRunningTime="2026-10-30 18:53:24.861077152 +0000 UTC m=+991.439172845"
Oct 30 18:53:25.836376 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:25.836343    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:25.838743 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:25.838718    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:26.727350 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:26.727308    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:53:32.849759 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:32.849727    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:53:43.035523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.035488    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:53:43.052174 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.052142    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:53:43.052334 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.052271    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.203419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.203370    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.203419 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.203423    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.203633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.203469    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.203633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.203501    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgnm\" (UniqueName: \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.203633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.203508    4942 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") DevicePath \"csi-421b7f3edd787477bd15de95fc25eab8d331b23e04a7494920f903dc8339453a\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.207354 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.207333    4942 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") DevicePath \"csi-421b7f3edd787477bd15de95fc25eab8d331b23e04a7494920f903dc8339453a\"" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.221621 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.221590    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.304754 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.304649    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.304754 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.304715    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgnm\" (UniqueName: \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.304997 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.304780    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.305337 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.305315    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.307165 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.307146    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.318082 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.318060    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgnm\" (UniqueName: \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\") pod \"proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") " pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.370063 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.369987    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.653824 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.653743    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:53:43.656657 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:53:43.656630    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998113d7_f225_4580_a707_3ecd2d6276de.slice/crio-6f637706e6f899b2c3e0e9f6bbab9ce4d76c2e6b461b62c6a7ae479f93df96b4 WatchSource:0}: Error finding container 6f637706e6f899b2c3e0e9f6bbab9ce4d76c2e6b461b62c6a7ae479f93df96b4: Status 404 returned error can't find the container with id 6f637706e6f899b2c3e0e9f6bbab9ce4d76c2e6b461b62c6a7ae479f93df96b4
Oct 30 18:53:43.916568 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916481    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerStarted","Data":"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"}
Oct 30 18:53:43.916716 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916588    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.916716 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916636    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerStarted","Data":"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"}
Oct 30 18:53:43.916716 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916677    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:43.916716 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916708    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerStarted","Data":"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"}
Oct 30 18:53:43.916857 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.916739    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerStarted","Data":"6f637706e6f899b2c3e0e9f6bbab9ce4d76c2e6b461b62c6a7ae479f93df96b4"}
Oct 30 18:53:43.917833 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.917806    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 30 18:53:43.945519 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:43.945455    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podStartSLOduration=1.945435459 podStartE2EDuration="1.945435459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:43.943340798 +0000 UTC m=+1010.521436500" watchObservedRunningTime="2026-10-30 18:53:43.945435459 +0000 UTC m=+1010.523531152"
Oct 30 18:53:44.921774 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:44.921742    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:44.922906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:44.922887    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:44.923323 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:44.923308    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:51.933975 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:51.933940    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:53:52.013942 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.013904    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:53:52.014289 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.014258    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kserve-container" containerID="cri-o://30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a" gracePeriod=30
Oct 30 18:53:52.014406 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.014309    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" containerID="cri-o://9b7b725e91be2eb57dc551f3dfa5fea469f50107f1ed29519bce552e1f40e779" gracePeriod=30
Oct 30 18:53:52.845857 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.845814    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:53:52.956588 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.956554    4942 generic.go:356] "Generic (PLEG): container finished" podID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerID="9b7b725e91be2eb57dc551f3dfa5fea469f50107f1ed29519bce552e1f40e779" exitCode=0
Oct 30 18:53:52.956973 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:52.956624    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerDied","Data":"9b7b725e91be2eb57dc551f3dfa5fea469f50107f1ed29519bce552e1f40e779"}
Oct 30 18:53:56.724128 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:53:56.724093    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:53:57.846449 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:53:57.846409    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:54:02.846328 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:02.846290    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:54:02.846725 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:02.846420    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:54:07.846248 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:07.846154    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:54:12.846254 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:12.846202    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:54:17.846532 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:17.846467    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 18:54:22.083234 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:22.083169    4942 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:54:22.083234 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:22.083203    4942 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-conmon-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:54:22.083822 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:22.083203    4942 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-conmon-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:54:22.089871 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:22.089811    4942 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48bc51_b173_49c2_a47e_5ea382d75ac5.slice/crio-conmon-30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:54:22.100970 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.100925    4942 generic.go:356] "Generic (PLEG): container finished" podID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerID="30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a" exitCode=137
Oct 30 18:54:22.101222 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.101100    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerDied","Data":"30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a"}
Oct 30 18:54:22.243954 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.243904    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:54:22.346450 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.346342    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\") pod \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") "
Oct 30 18:54:22.346450 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.346422    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\") pod \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") "
Oct 30 18:54:22.346648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.346557    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") "
Oct 30 18:54:22.346648 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.346597    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\" (UniqueName: \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\") pod \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\" (UID: \"bc48bc51-b173-49c2-a47e-5ea382d75ac5\") "
Oct 30 18:54:22.346808 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.346780    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config" pod "bc48bc51-b173-49c2-a47e-5ea382d75ac5" (UID: "bc48bc51-b173-49c2-a47e-5ea382d75ac5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:22.348982 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.348927    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls" pod "bc48bc51-b173-49c2-a47e-5ea382d75ac5" (UID: "bc48bc51-b173-49c2-a47e-5ea382d75ac5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:22.349177 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.348999    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65" pod "bc48bc51-b173-49c2-a47e-5ea382d75ac5" (UID: "bc48bc51-b173-49c2-a47e-5ea382d75ac5"). InnerVolumeSpecName "kube-api-access-ttl65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:22.351121 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.351089    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e" pod "bc48bc51-b173-49c2-a47e-5ea382d75ac5" (UID: "bc48bc51-b173-49c2-a47e-5ea382d75ac5"). InnerVolumeSpecName "pvc-e065d52b-8017-4276-a0ef-037ba9569d45". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 18:54:22.447940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.447895    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48bc51-b173-49c2-a47e-5ea382d75ac5-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:22.447940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.447936    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:22.447940 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:22.447947    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttl65\" (UniqueName: \"kubernetes.io/projected/bc48bc51-b173-49c2-a47e-5ea382d75ac5-kube-api-access-ttl65\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.108216 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.108175    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n" event={"ID":"bc48bc51-b173-49c2-a47e-5ea382d75ac5","Type":"ContainerDied","Data":"eb064531590e43d1a19c1f562904a8a48b2864683a6c50da6f5162958cbd2497"}
Oct 30 18:54:23.108216 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.108204    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"
Oct 30 18:54:23.108755 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.108247    4942 scope.go:122] "RemoveContainer" containerID="9b7b725e91be2eb57dc551f3dfa5fea469f50107f1ed29519bce552e1f40e779"
Oct 30 18:54:23.118384 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.118359    4942 scope.go:122] "RemoveContainer" containerID="30076915f0a460a37e8ce6023b789f8e8bd7ee2f9c8503c612bf7383b4d2e51a"
Oct 30 18:54:23.138495 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.138453    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:54:23.141652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.141604    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-6bd4448555-ltp7n"]
Oct 30 18:54:23.544223 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.544178    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" path="/var/lib/kubelet/pods/bc48bc51-b173-49c2-a47e-5ea382d75ac5/volumes"
Oct 30 18:54:23.705365 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.705321    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-3deaac58e4_proxy-routing-b24077bc-sink-885bf9646-sqxxq_0443b9b1-b318-4f22-8597-2b4d8e564310/kserve-container/0.log"
Oct 30 18:54:23.824401 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:23.824292    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:54:24.002763 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.002724    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:54:24.003224 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.003183    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kserve-container" containerID="cri-o://4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b" gracePeriod=30
Oct 30 18:54:24.003390 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.003211    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent" containerID="cri-o://a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee" gracePeriod=30
Oct 30 18:54:24.003390 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.003227    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" containerID="cri-o://926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c" gracePeriod=30
Oct 30 18:54:24.115754 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.115633    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" podUID="0443b9b1-b318-4f22-8597-2b4d8e564310" containerName="kserve-container" containerID="cri-o://d94b7c8faafdf63894db2ce38bf351f08ef8c4f90d56b76a5138acf6d44141d3" gracePeriod=30
Oct 30 18:54:24.922197 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:24.922153    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:25.123334 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:25.123293    4942 generic.go:356] "Generic (PLEG): container finished" podID="998113d7-f225-4580-a707-3ecd2d6276de" containerID="926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c" exitCode=0
Oct 30 18:54:25.123747 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:25.123352    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerDied","Data":"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"}
Oct 30 18:54:26.727699 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:26.727660    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:54:26.930254 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:26.930207    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:31.930319 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:31.930279    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:34.922253 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:34.922202    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:36.929735 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:36.929688    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:36.930249 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:36.929806    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:54:41.930201 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:41.930155    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:44.923126 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:44.923077    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:44.923585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:44.923222    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:54:46.929855 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:46.929810    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:51.930364 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:51.930314    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 18:54:54.202855 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.202829    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:54:54.253805 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.253772    4942 generic.go:356] "Generic (PLEG): container finished" podID="0443b9b1-b318-4f22-8597-2b4d8e564310" containerID="d94b7c8faafdf63894db2ce38bf351f08ef8c4f90d56b76a5138acf6d44141d3" exitCode=137
Oct 30 18:54:54.253933 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.253813    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" event={"ID":"0443b9b1-b318-4f22-8597-2b4d8e564310","Type":"ContainerDied","Data":"d94b7c8faafdf63894db2ce38bf351f08ef8c4f90d56b76a5138acf6d44141d3"}
Oct 30 18:54:54.256227 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256200    4942 generic.go:356] "Generic (PLEG): container finished" podID="998113d7-f225-4580-a707-3ecd2d6276de" containerID="a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee" exitCode=137
Oct 30 18:54:54.256227 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256227    4942 generic.go:356] "Generic (PLEG): container finished" podID="998113d7-f225-4580-a707-3ecd2d6276de" containerID="4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b" exitCode=137
Oct 30 18:54:54.256418 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256289    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerDied","Data":"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"}
Oct 30 18:54:54.256418 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256335    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"
Oct 30 18:54:54.256418 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256356    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerDied","Data":"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"}
Oct 30 18:54:54.256418 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256393    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j" event={"ID":"998113d7-f225-4580-a707-3ecd2d6276de","Type":"ContainerDied","Data":"6f637706e6f899b2c3e0e9f6bbab9ce4d76c2e6b461b62c6a7ae479f93df96b4"}
Oct 30 18:54:54.256613 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.256429    4942 scope.go:122] "RemoveContainer" containerID="a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"
Oct 30 18:54:54.264551 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.264532    4942 scope.go:122] "RemoveContainer" containerID="926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"
Oct 30 18:54:54.272757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.272734    4942 scope.go:122] "RemoveContainer" containerID="4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"
Oct 30 18:54:54.281128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281110    4942 scope.go:122] "RemoveContainer" containerID="a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"
Oct 30 18:54:54.281421 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:54.281393    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee\": container with ID starting with a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee not found: ID does not exist" containerID="a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"
Oct 30 18:54:54.281493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281432    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"} err="failed to get container status \"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee\": rpc error: code = NotFound desc = could not find container \"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee\": container with ID starting with a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee not found: ID does not exist"
Oct 30 18:54:54.281493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281450    4942 scope.go:122] "RemoveContainer" containerID="926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"
Oct 30 18:54:54.281693 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:54.281678    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c\": container with ID starting with 926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c not found: ID does not exist" containerID="926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"
Oct 30 18:54:54.281736 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281698    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"} err="failed to get container status \"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c\": rpc error: code = NotFound desc = could not find container \"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c\": container with ID starting with 926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c not found: ID does not exist"
Oct 30 18:54:54.281736 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281713    4942 scope.go:122] "RemoveContainer" containerID="4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"
Oct 30 18:54:54.281938 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:54.281924    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b\": container with ID starting with 4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b not found: ID does not exist" containerID="4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"
Oct 30 18:54:54.281978 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281942    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"} err="failed to get container status \"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b\": rpc error: code = NotFound desc = could not find container \"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b\": container with ID starting with 4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b not found: ID does not exist"
Oct 30 18:54:54.281978 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.281954    4942 scope.go:122] "RemoveContainer" containerID="a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"
Oct 30 18:54:54.282251 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.282227    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee"} err="failed to get container status \"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee\": rpc error: code = NotFound desc = could not find container \"a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee\": container with ID starting with a1d0d2a5bf8861e3c41e78f2696835ba04c747628b3fdd9287d42daab21763ee not found: ID does not exist"
Oct 30 18:54:54.282295 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.282253    4942 scope.go:122] "RemoveContainer" containerID="926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"
Oct 30 18:54:54.282486 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.282468    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c"} err="failed to get container status \"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c\": rpc error: code = NotFound desc = could not find container \"926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c\": container with ID starting with 926fecfe255064f2922ed75f8cb375f99376d1d0da7ddbb50a72a2e13cc2985c not found: ID does not exist"
Oct 30 18:54:54.282536 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.282495    4942 scope.go:122] "RemoveContainer" containerID="4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"
Oct 30 18:54:54.282739 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.282719    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b"} err="failed to get container status \"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b\": rpc error: code = NotFound desc = could not find container \"4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b\": container with ID starting with 4ad35224e2e438bd22bf0a26cb725c9a336109777da593a71ea2d6ca0d73144b not found: ID does not exist"
Oct 30 18:54:54.306460 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.306439    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:54:54.312767 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.312752    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") pod \"998113d7-f225-4580-a707-3ecd2d6276de\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") "
Oct 30 18:54:54.312829 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.312814    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\" (UniqueName: \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\") pod \"998113d7-f225-4580-a707-3ecd2d6276de\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") "
Oct 30 18:54:54.312885 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.312869    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\" (UniqueName: \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\") pod \"998113d7-f225-4580-a707-3ecd2d6276de\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") "
Oct 30 18:54:54.312954 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.312939    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\") pod \"998113d7-f225-4580-a707-3ecd2d6276de\" (UID: \"998113d7-f225-4580-a707-3ecd2d6276de\") "
Oct 30 18:54:54.313347 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.313322    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config" pod "998113d7-f225-4580-a707-3ecd2d6276de" (UID: "998113d7-f225-4580-a707-3ecd2d6276de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:54.314920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.314899    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls" pod "998113d7-f225-4580-a707-3ecd2d6276de" (UID: "998113d7-f225-4580-a707-3ecd2d6276de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:54.315045 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.314940    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm" pod "998113d7-f225-4580-a707-3ecd2d6276de" (UID: "998113d7-f225-4580-a707-3ecd2d6276de"). InnerVolumeSpecName "kube-api-access-gpgnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:54.315401 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.315381    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e" pod "998113d7-f225-4580-a707-3ecd2d6276de" (UID: "998113d7-f225-4580-a707-3ecd2d6276de"). InnerVolumeSpecName "pvc-e065d52b-8017-4276-a0ef-037ba9569d45". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 18:54:54.414308 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.414284    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\" (UniqueName: \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\") pod \"0443b9b1-b318-4f22-8597-2b4d8e564310\" (UID: \"0443b9b1-b318-4f22-8597-2b4d8e564310\") "
Oct 30 18:54:54.414428 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.414416    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgnm\" (UniqueName: \"kubernetes.io/projected/998113d7-f225-4580-a707-3ecd2d6276de-kube-api-access-gpgnm\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.414467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.414431    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/998113d7-f225-4580-a707-3ecd2d6276de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.414467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.414459    4942 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" "
Oct 30 18:54:54.414529 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.414473    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/998113d7-f225-4580-a707-3ecd2d6276de-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.416278 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.416254    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4" pod "0443b9b1-b318-4f22-8597-2b4d8e564310" (UID: "0443b9b1-b318-4f22-8597-2b4d8e564310"). InnerVolumeSpecName "kube-api-access-bkdt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:54.424304 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.424291    4942 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e065d52b-8017-4276-a0ef-037ba9569d45" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e") on node "ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:54:54.514897 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.514816    4942 reconciler_common.go:299] "Volume detached for volume \"pvc-e065d52b-8017-4276-a0ef-037ba9569d45\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0138bd81b10b9c16e\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"csi-421b7f3edd787477bd15de95fc25eab8d331b23e04a7494920f903dc8339453a\""
Oct 30 18:54:54.514897 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.514847    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdt4\" (UniqueName: \"kubernetes.io/projected/0443b9b1-b318-4f22-8597-2b4d8e564310-kube-api-access-bkdt4\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.585434 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.585402    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:54:54.598638 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.598612    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-predictor-78584f9dc4-b6m4j"]
Oct 30 18:54:54.863605 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863533    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:54:54.863885 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863867    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0443b9b1-b318-4f22-8597-2b4d8e564310" containerName="kserve-container"
Oct 30 18:54:54.863885 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863881    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0443b9b1-b318-4f22-8597-2b4d8e564310" containerName="kserve-container"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863900    4942 cpu_manager.go:414] "containerMap: removing container" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863905    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863913    4942 cpu_manager.go:414] "containerMap: removing container" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kserve-container"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863919    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kserve-container"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863931    4942 cpu_manager.go:414] "containerMap: removing container" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863938    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863955    4942 cpu_manager.go:414] "containerMap: removing container" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863960    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863969    4942 cpu_manager.go:414] "containerMap: removing container" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kserve-container"
Oct 30 18:54:54.864123 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.863974    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kserve-container"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864273    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kserve-container"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864292    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="0443b9b1-b318-4f22-8597-2b4d8e564310" containerName="kserve-container"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864303    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864310    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc48bc51-b173-49c2-a47e-5ea382d75ac5" containerName="kserve-container"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864326    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="kube-rbac-proxy"
Oct 30 18:54:54.864624 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.864334    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="998113d7-f225-4580-a707-3ecd2d6276de" containerName="agent"
Oct 30 18:54:54.867574 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.867554    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:54.879294 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:54.879269    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:54:55.018039 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.018000    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddsm\" (UniqueName: \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\") pod \"proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw\" (UID: \"2addbbe2-8a47-47c4-b08f-a1e014d03027\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:55.119436 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.119345    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddsm\" (UniqueName: \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\") pod \"proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw\" (UID: \"2addbbe2-8a47-47c4-b08f-a1e014d03027\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:55.134310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.134279    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddsm\" (UniqueName: \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\") pod \"proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw\" (UID: \"2addbbe2-8a47-47c4-b08f-a1e014d03027\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:55.179918 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.179888    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:55.263356 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.263326    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq" event={"ID":"0443b9b1-b318-4f22-8597-2b4d8e564310","Type":"ContainerDied","Data":"80890433c03fb88bc50b2ae083ccc86c7c080d395c6fb473dc5bb40c4014e90a"}
Oct 30 18:54:55.263705 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.263388    4942 scope.go:122] "RemoveContainer" containerID="d94b7c8faafdf63894db2ce38bf351f08ef8c4f90d56b76a5138acf6d44141d3"
Oct 30 18:54:55.263705 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.263344    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"
Oct 30 18:54:55.286852 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.286826    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:54:55.290996 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.290977    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-b24077bc-sink-885bf9646-sqxxq"]
Oct 30 18:54:55.458116 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.458094    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:54:55.462172 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:54:55.462147    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2addbbe2_8a47_47c4_b08f_a1e014d03027.slice/crio-22af60258f701942651e7188751684eb8f9fa3414f0fc410aeac2f75986d440f WatchSource:0}: Error finding container 22af60258f701942651e7188751684eb8f9fa3414f0fc410aeac2f75986d440f: Status 404 returned error can't find the container with id 22af60258f701942651e7188751684eb8f9fa3414f0fc410aeac2f75986d440f
Oct 30 18:54:55.541641 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.541618    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0443b9b1-b318-4f22-8597-2b4d8e564310" path="/var/lib/kubelet/pods/0443b9b1-b318-4f22-8597-2b4d8e564310/volumes"
Oct 30 18:54:55.542003 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:55.541990    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998113d7-f225-4580-a707-3ecd2d6276de" path="/var/lib/kubelet/pods/998113d7-f225-4580-a707-3ecd2d6276de/volumes"
Oct 30 18:54:56.271293 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:56.271254    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" event={"ID":"2addbbe2-8a47-47c4-b08f-a1e014d03027","Type":"ContainerStarted","Data":"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"}
Oct 30 18:54:56.271679 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:56.271311    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" event={"ID":"2addbbe2-8a47-47c4-b08f-a1e014d03027","Type":"ContainerStarted","Data":"22af60258f701942651e7188751684eb8f9fa3414f0fc410aeac2f75986d440f"}
Oct 30 18:54:56.271679 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:56.271470    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:56.273586 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:56.273564    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:54:56.292477 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:56.292435    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" podStartSLOduration=2.292425338 podStartE2EDuration="2.292425338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:56.291001747 +0000 UTC m=+1082.869097441" watchObservedRunningTime="2026-10-30 18:54:56.292425338 +0000 UTC m=+1082.870521082"
Oct 30 18:54:56.723868 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:54:56.723833    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:54:59.308947 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.308907    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:54:59.313658 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.313639    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.319128 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.319107    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-d1388bbd-dockercfg-lmfmh\""
Oct 30 18:54:59.319250 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.319140    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-d1388bbd-predictor-serving-cert\""
Oct 30 18:54:59.322493 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.322464    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-d1388bbd-kube-rbac-proxy-sar-config\""
Oct 30 18:54:59.323961 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.323937    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:54:59.353652 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.353619    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.353812 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.353686    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.353812 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.353789    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.454284 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.454240    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.454479 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.454344    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.454479 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.454396    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.455077 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.455056    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.456732 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.456702    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:54:59.468768 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:54:59.468735    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.107857 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.107817    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.115779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.115751    4942 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") device path: \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.208736 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.208703    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.208849 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.208826    4942 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") DevicePath \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.212609 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.212592    4942 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") DevicePath \"csi-9c21e0415df39d6642b287a44d3b0cbd904679a399c379503076890a59df691c\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.380678 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.380609    4942 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3fb19be4b6e87f69efd60325d940c074f928b5824c88d9be948bc5eae9ac44ed/globalmount\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.394200 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.394174    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.529093 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.529045    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:06.809752 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:06.809685    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:55:06.812452 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:55:06.812423    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043f3948_e5ac_4764_99b8_fd0d0e532875.slice/crio-c5f97ac207030548b3d41ea7559c9506c330c74044970ad5979679fc56c36a8f WatchSource:0}: Error finding container c5f97ac207030548b3d41ea7559c9506c330c74044970ad5979679fc56c36a8f: Status 404 returned error can't find the container with id c5f97ac207030548b3d41ea7559c9506c330c74044970ad5979679fc56c36a8f
Oct 30 18:55:07.320159 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.320124    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerStarted","Data":"03b15dd8b2eab4ade0711845125d541098804b5906bfd53c3ac9404dd4fd2be6"}
Oct 30 18:55:07.320587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.320180    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerStarted","Data":"9abdb64038b6a158d1d3a15754d3f4bb0ba07e662eca5bd0a64bfd846fd343fd"}
Oct 30 18:55:07.320587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.320199    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerStarted","Data":"c5f97ac207030548b3d41ea7559c9506c330c74044970ad5979679fc56c36a8f"}
Oct 30 18:55:07.320587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.320221    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:07.320587 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.320240    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:07.322730 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.322707    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:07.345828 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:07.345787    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podStartSLOduration=10.34577618 podStartE2EDuration="10.34577618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:07.345589375 +0000 UTC m=+1093.923685068" watchObservedRunningTime="2026-10-30 18:55:07.34577618 +0000 UTC m=+1093.923871866"
Oct 30 18:55:14.333875 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:14.333845    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:17.380804 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.380774    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:55:17.384521 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.384503    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.398133 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.398108    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:55:17.500582 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.500548    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.500753 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.500590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhv5g\" (UniqueName: \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.500753 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.500630    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.500753 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.500661    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.500753 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.500684    4942 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") DevicePath \"csi-9c21e0415df39d6642b287a44d3b0cbd904679a399c379503076890a59df691c\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.504608 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.504588    4942 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") DevicePath \"csi-9c21e0415df39d6642b287a44d3b0cbd904679a399c379503076890a59df691c\"" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.516583 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.516557    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.601448 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.601419    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhv5g\" (UniqueName: \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.601597 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.601469    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.601597 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.601502    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.602353 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.602329    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.603779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.603760    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.614774 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.614754    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhv5g\" (UniqueName: \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\") pod \"proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") " pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.701449 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.701425    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:17.983133 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:17.983101    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:55:17.985589 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:55:17.985564    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1590f643_ebee_4db0_8f87_24ca1f626cf7.slice/crio-236c7cb5c262f40cb7edf9b35f4ce1b381e46fa9d6577b4005e2538e4df10122 WatchSource:0}: Error finding container 236c7cb5c262f40cb7edf9b35f4ce1b381e46fa9d6577b4005e2538e4df10122: Status 404 returned error can't find the container with id 236c7cb5c262f40cb7edf9b35f4ce1b381e46fa9d6577b4005e2538e4df10122
Oct 30 18:55:18.371142 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371052    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerStarted","Data":"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"}
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371162    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371190    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerStarted","Data":"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"}
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371211    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerStarted","Data":"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"}
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371247    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371272    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerStarted","Data":"236c7cb5c262f40cb7edf9b35f4ce1b381e46fa9d6577b4005e2538e4df10122"}
Oct 30 18:55:18.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.371298    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:18.373505 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.373479    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:18.373915 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.373903    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:18.397738 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:18.397693    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podStartSLOduration=1.397681984 podStartE2EDuration="1.397681984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:18.396379493 +0000 UTC m=+1104.974475185" watchObservedRunningTime="2026-10-30 18:55:18.397681984 +0000 UTC m=+1104.975777675"
Oct 30 18:55:25.385204 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:25.385173    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:55:25.451143 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:25.451111    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:55:25.451360 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:25.451335    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kserve-container" containerID="cri-o://9abdb64038b6a158d1d3a15754d3f4bb0ba07e662eca5bd0a64bfd846fd343fd" gracePeriod=30
Oct 30 18:55:25.451423 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:25.451368    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" containerID="cri-o://03b15dd8b2eab4ade0711845125d541098804b5906bfd53c3ac9404dd4fd2be6" gracePeriod=30
Oct 30 18:55:26.409871 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:26.409838    4942 generic.go:356] "Generic (PLEG): container finished" podID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerID="03b15dd8b2eab4ade0711845125d541098804b5906bfd53c3ac9404dd4fd2be6" exitCode=0
Oct 30 18:55:26.410280 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:26.409911    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerDied","Data":"03b15dd8b2eab4ade0711845125d541098804b5906bfd53c3ac9404dd4fd2be6"}
Oct 30 18:55:26.726559 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:55:26.726519    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:55:29.330181 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:29.330139    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:34.330410 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:34.330312    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:39.329925 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:39.329877    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:39.330448 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:39.330054    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:44.330063 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:44.330002    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:49.330172 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:49.330126    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:54.330190 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:54.330144    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 18:55:55.537906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.537871    4942 generic.go:356] "Generic (PLEG): container finished" podID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerID="9abdb64038b6a158d1d3a15754d3f4bb0ba07e662eca5bd0a64bfd846fd343fd" exitCode=137
Oct 30 18:55:55.542301 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.542271    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerDied","Data":"9abdb64038b6a158d1d3a15754d3f4bb0ba07e662eca5bd0a64bfd846fd343fd"}
Oct 30 18:55:55.628272 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.628242    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:55.635191 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.635171    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\") pod \"043f3948-e5ac-4764-99b8-fd0d0e532875\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") "
Oct 30 18:55:55.635305 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.635215    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\") pod \"043f3948-e5ac-4764-99b8-fd0d0e532875\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") "
Oct 30 18:55:55.635305 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.635254    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\" (UniqueName: \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\") pod \"043f3948-e5ac-4764-99b8-fd0d0e532875\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") "
Oct 30 18:55:55.635529 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.635506    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config" pod "043f3948-e5ac-4764-99b8-fd0d0e532875" (UID: "043f3948-e5ac-4764-99b8-fd0d0e532875"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:55.637210 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.637189    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc" pod "043f3948-e5ac-4764-99b8-fd0d0e532875" (UID: "043f3948-e5ac-4764-99b8-fd0d0e532875"). InnerVolumeSpecName "kube-api-access-cwhcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:55.637342 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.637324    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls" pod "043f3948-e5ac-4764-99b8-fd0d0e532875" (UID: "043f3948-e5ac-4764-99b8-fd0d0e532875"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:55.735765 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.735678    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"043f3948-e5ac-4764-99b8-fd0d0e532875\" (UID: \"043f3948-e5ac-4764-99b8-fd0d0e532875\") "
Oct 30 18:55:55.735916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.735884    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043f3948-e5ac-4764-99b8-fd0d0e532875-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:55.735916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.735899    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:55.735916 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.735909    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/043f3948-e5ac-4764-99b8-fd0d0e532875-kube-api-access-cwhcc\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:55.738242 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:55.738215    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198" pod "043f3948-e5ac-4764-99b8-fd0d0e532875" (UID: "043f3948-e5ac-4764-99b8-fd0d0e532875"). InnerVolumeSpecName "pvc-dc410158-ac29-4c0d-8155-a789f1c01e47". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 18:55:56.544561 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.544535    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"
Oct 30 18:55:56.544561 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.544551    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx" event={"ID":"043f3948-e5ac-4764-99b8-fd0d0e532875","Type":"ContainerDied","Data":"c5f97ac207030548b3d41ea7559c9506c330c74044970ad5979679fc56c36a8f"}
Oct 30 18:55:56.544985 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.544617    4942 scope.go:122] "RemoveContainer" containerID="03b15dd8b2eab4ade0711845125d541098804b5906bfd53c3ac9404dd4fd2be6"
Oct 30 18:55:56.553189 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.553169    4942 scope.go:122] "RemoveContainer" containerID="9abdb64038b6a158d1d3a15754d3f4bb0ba07e662eca5bd0a64bfd846fd343fd"
Oct 30 18:55:56.570802 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.570779    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:55:56.573485 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:56.573460    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-568868f8d7-2rxwx"]
Oct 30 18:55:56.725147 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:55:56.725069    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:55:57.542819 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:55:57.542786    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" path="/var/lib/kubelet/pods/043f3948-e5ac-4764-99b8-fd0d0e532875/volumes"
Oct 30 18:56:01.964664 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:01.964613    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-3deaac58e4_proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc_1590f643-ebee-4db0-8f87-24ca1f626cf7/kserve-container/0.log"
Oct 30 18:56:02.058230 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.058198    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:56:02.058420 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.058398    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" podUID="2addbbe2-8a47-47c4-b08f-a1e014d03027" containerName="kserve-container" containerID="cri-o://77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92" gracePeriod=30
Oct 30 18:56:02.158349 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.158316    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:56:02.158641 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.158611    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kserve-container" containerID="cri-o://61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec" gracePeriod=30
Oct 30 18:56:02.158772 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.158608    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent" containerID="cri-o://bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e" gracePeriod=30
Oct 30 18:56:02.158831 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.158620    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" containerID="cri-o://d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40" gracePeriod=30
Oct 30 18:56:02.572487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.572404    4942 generic.go:356] "Generic (PLEG): container finished" podID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerID="d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40" exitCode=0
Oct 30 18:56:02.572487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:02.572468    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerDied","Data":"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"}
Oct 30 18:56:05.380945 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:05.380905    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:08.372791 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:08.372745    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:56:10.380931 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:10.380891    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:15.380967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:15.380920    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:15.381370 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:15.381095    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:56:18.371784 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:18.371730    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:56:20.381271 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:20.381228    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:25.381578 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:25.381536    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:26.724750 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:26.724715    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:56:28.372256 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:28.372215    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:56:28.372682 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:28.372360    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:56:30.381342 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:30.381299    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 18:56:32.251507 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.251481    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:56:32.297150 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.297126    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\" (UniqueName: \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\") pod \"2addbbe2-8a47-47c4-b08f-a1e014d03027\" (UID: \"2addbbe2-8a47-47c4-b08f-a1e014d03027\") "
Oct 30 18:56:32.299039 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.298993    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm" pod "2addbbe2-8a47-47c4-b08f-a1e014d03027" (UID: "2addbbe2-8a47-47c4-b08f-a1e014d03027"). InnerVolumeSpecName "kube-api-access-dddsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:32.333242 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.333221    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:56:32.398085 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.397984    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\") pod \"1590f643-ebee-4db0-8f87-24ca1f626cf7\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") "
Oct 30 18:56:32.398085 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.398056    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\" (UniqueName: \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\") pod \"1590f643-ebee-4db0-8f87-24ca1f626cf7\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") "
Oct 30 18:56:32.398311 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.398156    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") pod \"1590f643-ebee-4db0-8f87-24ca1f626cf7\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") "
Oct 30 18:56:32.398311 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.398204    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\") pod \"1590f643-ebee-4db0-8f87-24ca1f626cf7\" (UID: \"1590f643-ebee-4db0-8f87-24ca1f626cf7\") "
Oct 30 18:56:32.398467 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.398416    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddsm\" (UniqueName: \"kubernetes.io/projected/2addbbe2-8a47-47c4-b08f-a1e014d03027-kube-api-access-dddsm\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:32.398609 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.398566    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config" pod "1590f643-ebee-4db0-8f87-24ca1f626cf7" (UID: "1590f643-ebee-4db0-8f87-24ca1f626cf7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:32.400220 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.400192    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g" pod "1590f643-ebee-4db0-8f87-24ca1f626cf7" (UID: "1590f643-ebee-4db0-8f87-24ca1f626cf7"). InnerVolumeSpecName "kube-api-access-lhv5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:32.400313 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.400248    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls" pod "1590f643-ebee-4db0-8f87-24ca1f626cf7" (UID: "1590f643-ebee-4db0-8f87-24ca1f626cf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:32.400797 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.400779    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198" pod "1590f643-ebee-4db0-8f87-24ca1f626cf7" (UID: "1590f643-ebee-4db0-8f87-24ca1f626cf7"). InnerVolumeSpecName "pvc-dc410158-ac29-4c0d-8155-a789f1c01e47". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 18:56:32.498989 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.498960    4942 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" "
Oct 30 18:56:32.498989 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.498986    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:32.499175 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.498997    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590f643-ebee-4db0-8f87-24ca1f626cf7-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:32.499175 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.499028    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhv5g\" (UniqueName: \"kubernetes.io/projected/1590f643-ebee-4db0-8f87-24ca1f626cf7-kube-api-access-lhv5g\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:32.509142 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.509124    4942 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-dc410158-ac29-4c0d-8155-a789f1c01e47" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198") on node "ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:56:32.599812 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.599733    4942 reconciler_common.go:299] "Volume detached for volume \"pvc-dc410158-ac29-4c0d-8155-a789f1c01e47\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e8e005d4bf4d3198\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"csi-9c21e0415df39d6642b287a44d3b0cbd904679a399c379503076890a59df691c\""
Oct 30 18:56:32.696688 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.696657    4942 generic.go:356] "Generic (PLEG): container finished" podID="2addbbe2-8a47-47c4-b08f-a1e014d03027" containerID="77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92" exitCode=137
Oct 30 18:56:32.696834 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.696701    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"
Oct 30 18:56:32.696834 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.696730    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" event={"ID":"2addbbe2-8a47-47c4-b08f-a1e014d03027","Type":"ContainerDied","Data":"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"}
Oct 30 18:56:32.696834 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.696777    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw" event={"ID":"2addbbe2-8a47-47c4-b08f-a1e014d03027","Type":"ContainerDied","Data":"22af60258f701942651e7188751684eb8f9fa3414f0fc410aeac2f75986d440f"}
Oct 30 18:56:32.696834 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.696803    4942 scope.go:122] "RemoveContainer" containerID="77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"
Oct 30 18:56:32.698749 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698723    4942 generic.go:356] "Generic (PLEG): container finished" podID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerID="bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e" exitCode=0
Oct 30 18:56:32.698846 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698752    4942 generic.go:356] "Generic (PLEG): container finished" podID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerID="61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec" exitCode=137
Oct 30 18:56:32.698846 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698771    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"
Oct 30 18:56:32.698926 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698775    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerDied","Data":"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"}
Oct 30 18:56:32.698926 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698904    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerDied","Data":"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"}
Oct 30 18:56:32.699037 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.698944    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc" event={"ID":"1590f643-ebee-4db0-8f87-24ca1f626cf7","Type":"ContainerDied","Data":"236c7cb5c262f40cb7edf9b35f4ce1b381e46fa9d6577b4005e2538e4df10122"}
Oct 30 18:56:32.706817 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.706795    4942 scope.go:122] "RemoveContainer" containerID="77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"
Oct 30 18:56:32.707133 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:32.707112    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92\": container with ID starting with 77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92 not found: ID does not exist" containerID="77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"
Oct 30 18:56:32.707211 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.707140    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92"} err="failed to get container status \"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92\": rpc error: code = NotFound desc = could not find container \"77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92\": container with ID starting with 77b5078a44f90e3cdee908c6a301a0345fb3470dae8dd864e31efc8b9c4b9f92 not found: ID does not exist"
Oct 30 18:56:32.707211 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.707158    4942 scope.go:122] "RemoveContainer" containerID="bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"
Oct 30 18:56:32.714689 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.714668    4942 scope.go:122] "RemoveContainer" containerID="d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"
Oct 30 18:56:32.721790 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.721769    4942 scope.go:122] "RemoveContainer" containerID="61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"
Oct 30 18:56:32.724125 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.724106    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:56:32.729119 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729104    4942 scope.go:122] "RemoveContainer" containerID="bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"
Oct 30 18:56:32.729376 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:32.729359    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e\": container with ID starting with bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e not found: ID does not exist" containerID="bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"
Oct 30 18:56:32.729429 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729382    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"} err="failed to get container status \"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e\": rpc error: code = NotFound desc = could not find container \"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e\": container with ID starting with bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e not found: ID does not exist"
Oct 30 18:56:32.729429 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729397    4942 scope.go:122] "RemoveContainer" containerID="d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"
Oct 30 18:56:32.729529 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729514    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-sink-554cc4c7d9-jkrgw"]
Oct 30 18:56:32.729683 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:32.729664    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40\": container with ID starting with d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40 not found: ID does not exist" containerID="d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"
Oct 30 18:56:32.729724 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729686    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"} err="failed to get container status \"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40\": rpc error: code = NotFound desc = could not find container \"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40\": container with ID starting with d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40 not found: ID does not exist"
Oct 30 18:56:32.729724 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729699    4942 scope.go:122] "RemoveContainer" containerID="61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"
Oct 30 18:56:32.729924 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:32.729906    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec\": container with ID starting with 61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec not found: ID does not exist" containerID="61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"
Oct 30 18:56:32.729963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729927    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"} err="failed to get container status \"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec\": rpc error: code = NotFound desc = could not find container \"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec\": container with ID starting with 61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec not found: ID does not exist"
Oct 30 18:56:32.729963 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.729941    4942 scope.go:122] "RemoveContainer" containerID="bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"
Oct 30 18:56:32.730181 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.730161    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e"} err="failed to get container status \"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e\": rpc error: code = NotFound desc = could not find container \"bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e\": container with ID starting with bf3f5a9c1f99f0c5dc54c7de158655df0b518a6984369ac7c0f8d30b881c154e not found: ID does not exist"
Oct 30 18:56:32.730220 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.730182    4942 scope.go:122] "RemoveContainer" containerID="d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"
Oct 30 18:56:32.730414 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.730396    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40"} err="failed to get container status \"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40\": rpc error: code = NotFound desc = could not find container \"d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40\": container with ID starting with d8e33c35beef1a12163598d5445f3f46234c6d48b5d75757e4255aa470f27c40 not found: ID does not exist"
Oct 30 18:56:32.730454 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.730414    4942 scope.go:122] "RemoveContainer" containerID="61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"
Oct 30 18:56:32.730633 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.730615    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec"} err="failed to get container status \"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec\": rpc error: code = NotFound desc = could not find container \"61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec\": container with ID starting with 61d2265c1f7fc1bc7d889a53370195f16401eca99285bf3c9f5da45c0bffcaec not found: ID does not exist"
Oct 30 18:56:32.743267 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.743248    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:56:32.747974 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:32.747956    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-d1388bbd-predictor-5bfff8797f-x8crc"]
Oct 30 18:56:33.131794 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.131760    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:56:33.132318 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132297    4942 cpu_manager.go:414] "containerMap: removing container" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy"
Oct 30 18:56:33.132378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132322    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy"
Oct 30 18:56:33.132378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132363    4942 cpu_manager.go:414] "containerMap: removing container" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kserve-container"
Oct 30 18:56:33.132378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132372    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kserve-container"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132394    4942 cpu_manager.go:414] "containerMap: removing container" podUID="2addbbe2-8a47-47c4-b08f-a1e014d03027" containerName="kserve-container"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132402    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2addbbe2-8a47-47c4-b08f-a1e014d03027" containerName="kserve-container"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132414    4942 cpu_manager.go:414] "containerMap: removing container" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kserve-container"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132422    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kserve-container"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132433    4942 cpu_manager.go:414] "containerMap: removing container" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132440    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132449    4942 cpu_manager.go:414] "containerMap: removing container" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent"
Oct 30 18:56:33.132474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132456    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132544    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kserve-container"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132556    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="2addbbe2-8a47-47c4-b08f-a1e014d03027" containerName="kserve-container"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132565    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kserve-container"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132575    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="043f3948-e5ac-4764-99b8-fd0d0e532875" containerName="kube-rbac-proxy"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132585    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="agent"
Oct 30 18:56:33.132693 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.132597    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" containerName="kube-rbac-proxy"
Oct 30 18:56:33.137070 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.137046    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:33.143071 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.143046    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:56:33.152614 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.152591    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 18:56:33.152882 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.152857    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 18:56:33.152994 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.152881    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 18:56:33.204167 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.204067    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrmj\" (UniqueName: \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\") pod \"proxy-routing-c1e7fda2-sink-5844b58474-sjjt7\" (UID: \"a8c45bd5-ad84-4072-bf22-b1059ec02cdd\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:33.305247 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.305222    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrmj\" (UniqueName: \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\") pod \"proxy-routing-c1e7fda2-sink-5844b58474-sjjt7\" (UID: \"a8c45bd5-ad84-4072-bf22-b1059ec02cdd\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:33.318330 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.318306    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrmj\" (UniqueName: \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\") pod \"proxy-routing-c1e7fda2-sink-5844b58474-sjjt7\" (UID: \"a8c45bd5-ad84-4072-bf22-b1059ec02cdd\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:33.451437 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.451412    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:33.542282 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.542241    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1590f643-ebee-4db0-8f87-24ca1f626cf7" path="/var/lib/kubelet/pods/1590f643-ebee-4db0-8f87-24ca1f626cf7/volumes"
Oct 30 18:56:33.542731 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.542710    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2addbbe2-8a47-47c4-b08f-a1e014d03027" path="/var/lib/kubelet/pods/2addbbe2-8a47-47c4-b08f-a1e014d03027/volumes"
Oct 30 18:56:33.728074 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:33.727980    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:56:33.731279 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:56:33.731252    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c45bd5_ad84_4072_bf22_b1059ec02cdd.slice/crio-fe3d7aae671239142781b93aecd02365c6a350a446d313917b65f64ca28d9c09 WatchSource:0}: Error finding container fe3d7aae671239142781b93aecd02365c6a350a446d313917b65f64ca28d9c09: Status 404 returned error can't find the container with id fe3d7aae671239142781b93aecd02365c6a350a446d313917b65f64ca28d9c09
Oct 30 18:56:34.714086 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:34.714044    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" event={"ID":"a8c45bd5-ad84-4072-bf22-b1059ec02cdd","Type":"ContainerStarted","Data":"4eaa34bd450e2916ab667b5288f306834d1df9e354de447f040fcf964f8c4cc0"}
Oct 30 18:56:34.714563 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:34.714099    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" event={"ID":"a8c45bd5-ad84-4072-bf22-b1059ec02cdd","Type":"ContainerStarted","Data":"fe3d7aae671239142781b93aecd02365c6a350a446d313917b65f64ca28d9c09"}
Oct 30 18:56:34.714563 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:34.714216    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:34.716642 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:34.716619    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:56:34.736497 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:34.736457    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" podStartSLOduration=1.736446642 podStartE2EDuration="1.736446642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:34.735852936 +0000 UTC m=+1181.313948629" watchObservedRunningTime="2026-10-30 18:56:34.736446642 +0000 UTC m=+1181.314542346"
Oct 30 18:56:38.536065 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.536031    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:56:38.539776 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.539750    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.550661 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.550637    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-c1e7fda2-kube-rbac-proxy-sar-config\""
Oct 30 18:56:38.551561 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.551541    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-c1e7fda2-dockercfg-r7fsc\""
Oct 30 18:56:38.551762 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.551743    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"proxy-routing-c1e7fda2-predictor-serving-cert\""
Oct 30 18:56:38.553062 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.553042    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:56:38.647741 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.647708    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.647899 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.647851    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffvp\" (UniqueName: \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.647899 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.647891    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.748902 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.748871    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.749065 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.748928    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffvp\" (UniqueName: \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.749065 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.748954    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.749524 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.749493    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.751300 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.751281    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:38.762475 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:38.762445    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffvp\" (UniqueName: \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.634194 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.634161    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.645188 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.645166    4942 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") device path: \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.735359 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.735327    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.735523 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.735459    4942 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") DevicePath \"\"" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.739378 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.739360    4942 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") DevicePath \"csi-7204ae685add6638ecf82d6ef37a0ac4775f93ce37ff00712208ae0e9657baf5\"" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.907945 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.907876    4942 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e558404102d9f4321d76d60c74bcfc55ce5c626b79787676680df55e791da2d4/globalmount\"" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:48.921352 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:48.921326    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") " pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:49.055550 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.055510    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:49.337540 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.337506    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:56:49.341056 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:56:49.341029    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80263385_26d9_4ba5_aee5_0a4d392c7a6c.slice/crio-b6f594071b5dcb78ff86fd535f06728e1f7472912f53a6a94c47adecb7bb71c1 WatchSource:0}: Error finding container b6f594071b5dcb78ff86fd535f06728e1f7472912f53a6a94c47adecb7bb71c1: Status 404 returned error can't find the container with id b6f594071b5dcb78ff86fd535f06728e1f7472912f53a6a94c47adecb7bb71c1
Oct 30 18:56:49.778295 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.778257    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerStarted","Data":"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"}
Oct 30 18:56:49.778689 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.778319    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerStarted","Data":"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"}
Oct 30 18:56:49.778689 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.778344    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerStarted","Data":"b6f594071b5dcb78ff86fd535f06728e1f7472912f53a6a94c47adecb7bb71c1"}
Oct 30 18:56:49.778689 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.778421    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:49.802886 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:49.802839    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podStartSLOduration=14.802828518 podStartE2EDuration="14.802828518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:49.800938301 +0000 UTC m=+1196.379033995" watchObservedRunningTime="2026-10-30 18:56:49.802828518 +0000 UTC m=+1196.380924209"
Oct 30 18:56:50.783087 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:50.783054    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:50.785366 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:50.785338    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:56:53.640824 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:53.640791    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:56:53.642951 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:53.642929    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 18:56:56.726417 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:56:56.726373    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:56:57.795019 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:56:57.794985    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:57:26.729222 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:57:26.729173    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:57:36.552017 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.551929    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:57:36.552558 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.552246    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" podUID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" containerName="kserve-container" containerID="cri-o://4eaa34bd450e2916ab667b5288f306834d1df9e354de447f040fcf964f8c4cc0" gracePeriod=30
Oct 30 18:57:36.696546 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.696494    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:57:36.696872 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.696834    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kserve-container" containerID="cri-o://81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606" gracePeriod=30
Oct 30 18:57:36.697058 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.696877    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" containerID="cri-o://141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa" gracePeriod=30
Oct 30 18:57:36.980155 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.980108    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:57:36.984096 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.984067    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:36.991610 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.991570    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 18:57:36.991814 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.991636    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:57:36.991814 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.991711    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 18:57:36.991814 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.991724    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 18:57:36.995713 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.995674    4942 generic.go:356] "Generic (PLEG): container finished" podID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerID="141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa" exitCode=0
Oct 30 18:57:36.996080 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.995754    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerDied","Data":"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"}
Oct 30 18:57:36.996370 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.996346    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 18:57:36.996520 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.996504    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 18:57:36.996570 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:36.996535    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 30 18:57:37.007585 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.007544    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:57:37.059806 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.059701    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxqk\" (UniqueName: \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.060020 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.059808    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.060020 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.059832    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.060020 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.059863    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.060148 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.059998    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.160487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.160432    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.160487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.160482    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.160779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.160520    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.160779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.160548    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.160779 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.160579    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxqk\" (UniqueName: \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.161057 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.161028    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.161347 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.161319    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.161489 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.161463    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.164399 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.164355    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.178678 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.178635    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxqk\" (UniqueName: \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\") pod \"isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.308754 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.308717    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:57:37.608051 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.607921    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:57:37.611597 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:57:37.611558    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15300735_ba16_4551_8127_e19fbb249333.slice/crio-e09130ab4a608c1b49c2824c8a892cf724187da868bad197549407cfda7356dd WatchSource:0}: Error finding container e09130ab4a608c1b49c2824c8a892cf724187da868bad197549407cfda7356dd: Status 404 returned error can't find the container with id e09130ab4a608c1b49c2824c8a892cf724187da868bad197549407cfda7356dd
Oct 30 18:57:37.791640 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:37.791576    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:57:38.003247 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:38.003204    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerStarted","Data":"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"}
Oct 30 18:57:38.003495 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:38.003282    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerStarted","Data":"e09130ab4a608c1b49c2824c8a892cf724187da868bad197549407cfda7356dd"}
Oct 30 18:57:42.025946 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:42.025902    4942 generic.go:356] "Generic (PLEG): container finished" podID="15300735-ba16-4551-8127-e19fbb249333" containerID="7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810" exitCode=0
Oct 30 18:57:42.026395 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:42.025970    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerDied","Data":"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"}
Oct 30 18:57:42.791864 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:42.791450    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:57:47.791702 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:47.791652    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:57:47.792241 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:47.791785    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:57:52.792067 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:52.791657    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:57:56.730847 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:57:56.730802    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:57:57.791192 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:57:57.791133    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:58:02.791348 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:02.791294    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 18:58:05.171680 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:05.171646    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerStarted","Data":"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"}
Oct 30 18:58:05.172226 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:05.171719    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerStarted","Data":"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"}
Oct 30 18:58:05.172226 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:05.171938    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:58:05.199095 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:05.199033    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podStartSLOduration=2.321936038 podStartE2EDuration="29.199018902s" totalImagesPullingTime="22.541023159s" totalInitContainerRuntime="4.336059705s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:05.196484261 +0000 UTC m=+1271.774579966" watchObservedRunningTime="2026-10-30 18:58:05.199018902 +0000 UTC m=+1271.777114584"
Oct 30 18:58:06.176095 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:06.176062    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:58:06.923622 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:06.923591    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:58:07.027307 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.027267    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") pod \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") "
Oct 30 18:58:07.027524 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.027331    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\") pod \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") "
Oct 30 18:58:07.027628 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.027600    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\" (UniqueName: \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\") pod \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") "
Oct 30 18:58:07.027691 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.027657    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\") pod \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\" (UID: \"80263385-26d9-4ba5-aee5-0a4d392c7a6c\") "
Oct 30 18:58:07.027826 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.027796    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config" pod "80263385-26d9-4ba5-aee5-0a4d392c7a6c" (UID: "80263385-26d9-4ba5-aee5-0a4d392c7a6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:07.028201 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.028173    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:07.030109 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.030074    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp" pod "80263385-26d9-4ba5-aee5-0a4d392c7a6c" (UID: "80263385-26d9-4ba5-aee5-0a4d392c7a6c"). InnerVolumeSpecName "kube-api-access-pffvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:07.030240 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.030087    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls" pod "80263385-26d9-4ba5-aee5-0a4d392c7a6c" (UID: "80263385-26d9-4ba5-aee5-0a4d392c7a6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:07.030352 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.030274    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92" pod "80263385-26d9-4ba5-aee5-0a4d392c7a6c" (UID: "80263385-26d9-4ba5-aee5-0a4d392c7a6c"). InnerVolumeSpecName "pvc-53f50283-74b3-4bd3-978f-afa5534a77cf". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 18:58:07.128675 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.128611    4942 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" "
Oct 30 18:58:07.128675 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.128639    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffvp\" (UniqueName: \"kubernetes.io/projected/80263385-26d9-4ba5-aee5-0a4d392c7a6c-kube-api-access-pffvp\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:07.128675 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.128651    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80263385-26d9-4ba5-aee5-0a4d392c7a6c-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:07.139025 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.138985    4942 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-53f50283-74b3-4bd3-978f-afa5534a77cf" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92") on node "ip-10-0-134-34.us-east-2.compute.internal"
Oct 30 18:58:07.182035 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.181992    4942 generic.go:356] "Generic (PLEG): container finished" podID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerID="81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606" exitCode=137
Oct 30 18:58:07.182474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.182163    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerDied","Data":"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"}
Oct 30 18:58:07.182474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.182236    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"
Oct 30 18:58:07.182474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.182291    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw" event={"ID":"80263385-26d9-4ba5-aee5-0a4d392c7a6c","Type":"ContainerDied","Data":"b6f594071b5dcb78ff86fd535f06728e1f7472912f53a6a94c47adecb7bb71c1"}
Oct 30 18:58:07.182474 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.182337    4942 scope.go:122] "RemoveContainer" containerID="141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"
Oct 30 18:58:07.186931 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.186904    4942 generic.go:356] "Generic (PLEG): container finished" podID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" containerID="4eaa34bd450e2916ab667b5288f306834d1df9e354de447f040fcf964f8c4cc0" exitCode=137
Oct 30 18:58:07.187590 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.187568    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" event={"ID":"a8c45bd5-ad84-4072-bf22-b1059ec02cdd","Type":"ContainerDied","Data":"4eaa34bd450e2916ab667b5288f306834d1df9e354de447f040fcf964f8c4cc0"}
Oct 30 18:58:07.195960 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.195938    4942 scope.go:122] "RemoveContainer" containerID="81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"
Oct 30 18:58:07.207806 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.207779    4942 scope.go:122] "RemoveContainer" containerID="141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"
Oct 30 18:58:07.208163 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:58:07.208137    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa\": container with ID starting with 141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa not found: ID does not exist" containerID="141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"
Oct 30 18:58:07.208487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.208170    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa"} err="failed to get container status \"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa\": rpc error: code = NotFound desc = could not find container \"141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa\": container with ID starting with 141c9a9707df3571544ec80d71e917cd5952c5ff8f1b48b1e8bdf98255800daa not found: ID does not exist"
Oct 30 18:58:07.208487 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.208190    4942 scope.go:122] "RemoveContainer" containerID="81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"
Oct 30 18:58:07.208563 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:58:07.208483    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606\": container with ID starting with 81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606 not found: ID does not exist" containerID="81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"
Oct 30 18:58:07.208563 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.208517    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606"} err="failed to get container status \"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606\": rpc error: code = NotFound desc = could not find container \"81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606\": container with ID starting with 81c4cafab15e827fba5dcd09d24814fafae1f3f1aeb0040bffefdf9d56242606 not found: ID does not exist"
Oct 30 18:58:07.229629 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.229598    4942 reconciler_common.go:299] "Volume detached for volume \"pvc-53f50283-74b3-4bd3-978f-afa5534a77cf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d0bd9e3b33bd4e92\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"csi-7204ae685add6638ecf82d6ef37a0ac4775f93ce37ff00712208ae0e9657baf5\""
Oct 30 18:58:07.242386 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.242355    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:58:07.281071 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.281031    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-predictor-98dfc7dd4-v4lkw"]
Oct 30 18:58:07.285343 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.285318    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:58:07.432408 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.432374    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\" (UniqueName: \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\") pod \"a8c45bd5-ad84-4072-bf22-b1059ec02cdd\" (UID: \"a8c45bd5-ad84-4072-bf22-b1059ec02cdd\") "
Oct 30 18:58:07.434465 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.434443    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj" pod "a8c45bd5-ad84-4072-bf22-b1059ec02cdd" (UID: "a8c45bd5-ad84-4072-bf22-b1059ec02cdd"). InnerVolumeSpecName "kube-api-access-sdrmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:07.533126 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.533086    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrmj\" (UniqueName: \"kubernetes.io/projected/a8c45bd5-ad84-4072-bf22-b1059ec02cdd-kube-api-access-sdrmj\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:07.541471 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:07.541442    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" path="/var/lib/kubelet/pods/80263385-26d9-4ba5-aee5-0a4d392c7a6c/volumes"
Oct 30 18:58:08.191924 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:08.191891    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"
Oct 30 18:58:08.192449 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:08.191916    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7" event={"ID":"a8c45bd5-ad84-4072-bf22-b1059ec02cdd","Type":"ContainerDied","Data":"fe3d7aae671239142781b93aecd02365c6a350a446d313917b65f64ca28d9c09"}
Oct 30 18:58:08.192449 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:08.192039    4942 scope.go:122] "RemoveContainer" containerID="4eaa34bd450e2916ab667b5288f306834d1df9e354de447f040fcf964f8c4cc0"
Oct 30 18:58:08.211287 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:08.211261    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:58:08.216870 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:08.216849    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/proxy-routing-c1e7fda2-sink-5844b58474-sjjt7"]
Oct 30 18:58:09.541232 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:09.541198    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" path="/var/lib/kubelet/pods/a8c45bd5-ad84-4072-bf22-b1059ec02cdd/volumes"
Oct 30 18:58:12.193111 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:12.193083    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:58:26.724284 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:58:26.724245    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:58:42.194144 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:42.194060    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 18:58:52.193965 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:58:52.193922    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 18:58:56.724294 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:58:56.724259    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:59:02.193864 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:02.193828    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 18:59:12.752907 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:12.752863    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:59:17.440481 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:17.440433    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:59:17.440967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:17.440757    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container" containerID="cri-o://8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3" gracePeriod=30
Oct 30 18:59:17.440967 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:17.440840    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kube-rbac-proxy" containerID="cri-o://ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c" gracePeriod=30
Oct 30 18:59:18.488634 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:18.488582    4942 generic.go:356] "Generic (PLEG): container finished" podID="15300735-ba16-4551-8127-e19fbb249333" containerID="ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c" exitCode=0
Oct 30 18:59:18.489083 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:18.488668    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerDied","Data":"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"}
Oct 30 18:59:22.188466 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:22.188407    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Oct 30 18:59:22.193833 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:22.193794    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 18:59:23.015511 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.015472    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:59:23.160606 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160560    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\") pod \"15300735-ba16-4551-8127-e19fbb249333\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") "
Oct 30 18:59:23.160798 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160630    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\" (UniqueName: \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\") pod \"15300735-ba16-4551-8127-e19fbb249333\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") "
Oct 30 18:59:23.160798 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160676    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\") pod \"15300735-ba16-4551-8127-e19fbb249333\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") "
Oct 30 18:59:23.160798 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160697    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\") pod \"15300735-ba16-4551-8127-e19fbb249333\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") "
Oct 30 18:59:23.160798 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160721    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\" (UniqueName: \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\") pod \"15300735-ba16-4551-8127-e19fbb249333\" (UID: \"15300735-ba16-4551-8127-e19fbb249333\") "
Oct 30 18:59:23.161130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.160981    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location" pod "15300735-ba16-4551-8127-e19fbb249333" (UID: "15300735-ba16-4551-8127-e19fbb249333"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:23.161130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.161092    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert" pod "15300735-ba16-4551-8127-e19fbb249333" (UID: "15300735-ba16-4551-8127-e19fbb249333"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:23.161303 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.161197    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config" pod "15300735-ba16-4551-8127-e19fbb249333" (UID: "15300735-ba16-4551-8127-e19fbb249333"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:23.163211 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.163173    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls" pod "15300735-ba16-4551-8127-e19fbb249333" (UID: "15300735-ba16-4551-8127-e19fbb249333"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:23.163371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.163342    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk" pod "15300735-ba16-4551-8127-e19fbb249333" (UID: "15300735-ba16-4551-8127-e19fbb249333"). InnerVolumeSpecName "kube-api-access-fmxqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:23.261602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.261505    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:23.261602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.261543    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15300735-ba16-4551-8127-e19fbb249333-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:23.261602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.261556    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15300735-ba16-4551-8127-e19fbb249333-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:23.261602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.261566    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15300735-ba16-4551-8127-e19fbb249333-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:23.261602 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.261575    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxqk\" (UniqueName: \"kubernetes.io/projected/15300735-ba16-4551-8127-e19fbb249333-kube-api-access-fmxqk\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:23.514136 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.514040    4942 generic.go:356] "Generic (PLEG): container finished" podID="15300735-ba16-4551-8127-e19fbb249333" containerID="8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3" exitCode=0
Oct 30 18:59:23.514136 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.514131    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerDied","Data":"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"}
Oct 30 18:59:23.514341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.514170    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"
Oct 30 18:59:23.514341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.514200    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5" event={"ID":"15300735-ba16-4551-8127-e19fbb249333","Type":"ContainerDied","Data":"e09130ab4a608c1b49c2824c8a892cf724187da868bad197549407cfda7356dd"}
Oct 30 18:59:23.514341 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.514240    4942 scope.go:122] "RemoveContainer" containerID="ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"
Oct 30 18:59:23.524356 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.524321    4942 scope.go:122] "RemoveContainer" containerID="8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"
Oct 30 18:59:23.534501 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.534474    4942 scope.go:122] "RemoveContainer" containerID="7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"
Oct 30 18:59:23.545769 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.545741    4942 scope.go:122] "RemoveContainer" containerID="ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"
Oct 30 18:59:23.545949 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.545919    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:59:23.546254 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:59:23.546204    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c\": container with ID starting with ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c not found: ID does not exist" containerID="ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"
Oct 30 18:59:23.546387 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.546243    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c"} err="failed to get container status \"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c\": rpc error: code = NotFound desc = could not find container \"ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c\": container with ID starting with ddf4722a6d25a6390a85f6ebaa738da8dd2d81253b8513dd3c80618c878bb06c not found: ID does not exist"
Oct 30 18:59:23.546387 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.546271    4942 scope.go:122] "RemoveContainer" containerID="8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"
Oct 30 18:59:23.546637 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:59:23.546612    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3\": container with ID starting with 8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3 not found: ID does not exist" containerID="8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"
Oct 30 18:59:23.546757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.546642    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3"} err="failed to get container status \"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3\": rpc error: code = NotFound desc = could not find container \"8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3\": container with ID starting with 8f24d661121451ed35622da17707096421062bc527ce71962ecdb12433f0e5b3 not found: ID does not exist"
Oct 30 18:59:23.546757 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.546661    4942 scope.go:122] "RemoveContainer" containerID="7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"
Oct 30 18:59:23.547153 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:59:23.547092    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810\": container with ID starting with 7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810 not found: ID does not exist" containerID="7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"
Oct 30 18:59:23.547153 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.547128    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810"} err="failed to get container status \"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810\": rpc error: code = NotFound desc = could not find container \"7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810\": container with ID starting with 7886cbfac4097c4894da58044541d87e86a2354ad3ad4096595c9fd3a8653810 not found: ID does not exist"
Oct 30 18:59:23.548926 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:23.548886    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-xgboost-v2-predictor-669cb4b7db-jrmh5"]
Oct 30 18:59:24.951937 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.951891    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952251    4942 cpu_manager.go:414] "containerMap: removing container" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="storage-initializer"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952263    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="storage-initializer"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952286    4942 cpu_manager.go:414] "containerMap: removing container" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952292    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952301    4942 cpu_manager.go:414] "containerMap: removing container" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kube-rbac-proxy"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952307    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kube-rbac-proxy"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952315    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952321    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952338    4942 cpu_manager.go:414] "containerMap: removing container" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952343    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kserve-container"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952350    4942 cpu_manager.go:414] "containerMap: removing container" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy"
Oct 30 18:59:24.952371 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952355    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy"
Oct 30 18:59:24.952766 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952421    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kserve-container"
Oct 30 18:59:24.952766 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952431    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="15300735-ba16-4551-8127-e19fbb249333" containerName="kube-rbac-proxy"
Oct 30 18:59:24.952766 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952438    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c45bd5-ad84-4072-bf22-b1059ec02cdd" containerName="kserve-container"
Oct 30 18:59:24.952766 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952443    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kserve-container"
Oct 30 18:59:24.952766 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.952449    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="80263385-26d9-4ba5-aee5-0a4d392c7a6c" containerName="kube-rbac-proxy"
Oct 30 18:59:24.957374 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.957343    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:24.975117 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.975080    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 18:59:24.977732 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.977699    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 18:59:24.977918 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.977741    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 18:59:24.977918 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.977741    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 30 18:59:24.978231 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.978209    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 18:59:24.978310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.978242    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 30 18:59:24.978310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.978258    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 18:59:24.978310 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:24.978283    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:59:25.077760 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.077712    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7zg\" (UniqueName: \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.077952 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.077770    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.077952 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.077842    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.077952 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.077891    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.078121 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.077952    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179091    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179130 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179131    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179170    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7zg\" (UniqueName: \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179201    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179338 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179217    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179719 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179684    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179944 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179923    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.179988 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.179934    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.181785 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.181755    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.192591 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.192563    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7zg\" (UniqueName: \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\") pod \"isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") " pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.276389 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.276281    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:25.542513 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.542433    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15300735-ba16-4551-8127-e19fbb249333" path="/var/lib/kubelet/pods/15300735-ba16-4551-8127-e19fbb249333/volumes"
Oct 30 18:59:25.561936 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:25.561903    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 18:59:25.565136 ip-10-0-134-34 kubenswrapper[4942]: W1030 18:59:25.565105    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616b23b6_9bb3_4465_9bb8_26c472f95f71.slice/crio-e071e03e99ec2c56d454a0ef7a48922cdc2ea6771c2ed1c4150a4d233081be56 WatchSource:0}: Error finding container e071e03e99ec2c56d454a0ef7a48922cdc2ea6771c2ed1c4150a4d233081be56: Status 404 returned error can't find the container with id e071e03e99ec2c56d454a0ef7a48922cdc2ea6771c2ed1c4150a4d233081be56
Oct 30 18:59:26.533462 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:26.533405    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerStarted","Data":"08a2aca11f3a09aa1de3db6dc6f3198fd6d769008cc18473ea025e5b41ee019d"}
Oct 30 18:59:26.533920 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:26.533494    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerStarted","Data":"e071e03e99ec2c56d454a0ef7a48922cdc2ea6771c2ed1c4150a4d233081be56"}
Oct 30 18:59:26.726130 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:59:26.726090    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 18:59:30.552407 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:30.552369    4942 generic.go:356] "Generic (PLEG): container finished" podID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerID="08a2aca11f3a09aa1de3db6dc6f3198fd6d769008cc18473ea025e5b41ee019d" exitCode=0
Oct 30 18:59:30.552913 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:30.552445    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerDied","Data":"08a2aca11f3a09aa1de3db6dc6f3198fd6d769008cc18473ea025e5b41ee019d"}
Oct 30 18:59:31.559111 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:31.559073    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerStarted","Data":"e2056f17fe4e3f7c1857fcc1c704f3a8c4ba0dd90fda44c1a4667a62df10b0be"}
Oct 30 18:59:31.559504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:31.559141    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerStarted","Data":"cb0e06f91518f54d2bb03225836ce2684934017acd6c56c11de91eb6900469a8"}
Oct 30 18:59:31.559504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:31.559342    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:31.559504 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:31.559387    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:31.589906 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:31.589849    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" podStartSLOduration=3.437189788 podStartE2EDuration="7.589835283s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.152645495s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:31.587363381 +0000 UTC m=+1358.165459201" watchObservedRunningTime="2026-10-30 18:59:31.589835283 +0000 UTC m=+1358.167930975"
Oct 30 18:59:37.568994 ip-10-0-134-34 kubenswrapper[4942]: I1030 18:59:37.568962    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 18:59:56.723729 ip-10-0-134-34 kubenswrapper[4942]: E1030 18:59:56.723695    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:00:07.629972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:07.629896    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 19:00:15.194633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.194601    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 19:00:15.195099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.194819    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kserve-container" containerID="cri-o://cb0e06f91518f54d2bb03225836ce2684934017acd6c56c11de91eb6900469a8" gracePeriod=30
Oct 30 19:00:15.195099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.194862    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kube-rbac-proxy" containerID="cri-o://e2056f17fe4e3f7c1857fcc1c704f3a8c4ba0dd90fda44c1a4667a62df10b0be" gracePeriod=30
Oct 30 19:00:15.755910 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.755877    4942 generic.go:356] "Generic (PLEG): container finished" podID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerID="e2056f17fe4e3f7c1857fcc1c704f3a8c4ba0dd90fda44c1a4667a62df10b0be" exitCode=0
Oct 30 19:00:15.755910 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.755905    4942 generic.go:356] "Generic (PLEG): container finished" podID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerID="cb0e06f91518f54d2bb03225836ce2684934017acd6c56c11de91eb6900469a8" exitCode=0
Oct 30 19:00:15.756114 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.755953    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerDied","Data":"e2056f17fe4e3f7c1857fcc1c704f3a8c4ba0dd90fda44c1a4667a62df10b0be"}
Oct 30 19:00:15.756114 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.756026    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerDied","Data":"cb0e06f91518f54d2bb03225836ce2684934017acd6c56c11de91eb6900469a8"}
Oct 30 19:00:15.831682 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.831659    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 19:00:15.892501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892474    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\") pod \"616b23b6-9bb3-4465-9bb8-26c472f95f71\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") "
Oct 30 19:00:15.892648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892531    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\" (UniqueName: \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\") pod \"616b23b6-9bb3-4465-9bb8-26c472f95f71\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") "
Oct 30 19:00:15.892648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892576    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\") pod \"616b23b6-9bb3-4465-9bb8-26c472f95f71\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") "
Oct 30 19:00:15.892648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892626    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\" (UniqueName: \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\") pod \"616b23b6-9bb3-4465-9bb8-26c472f95f71\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") "
Oct 30 19:00:15.892803 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892678    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\") pod \"616b23b6-9bb3-4465-9bb8-26c472f95f71\" (UID: \"616b23b6-9bb3-4465-9bb8-26c472f95f71\") "
Oct 30 19:00:15.892885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892857    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert" pod "616b23b6-9bb3-4465-9bb8-26c472f95f71" (UID: "616b23b6-9bb3-4465-9bb8-26c472f95f71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:15.892934 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.892914    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location" pod "616b23b6-9bb3-4465-9bb8-26c472f95f71" (UID: "616b23b6-9bb3-4465-9bb8-26c472f95f71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:15.893136 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.893112    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config" pod "616b23b6-9bb3-4465-9bb8-26c472f95f71" (UID: "616b23b6-9bb3-4465-9bb8-26c472f95f71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:15.894644 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.894618    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg" pod "616b23b6-9bb3-4465-9bb8-26c472f95f71" (UID: "616b23b6-9bb3-4465-9bb8-26c472f95f71"). InnerVolumeSpecName "kube-api-access-7l7zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:15.894738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.894641    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls" pod "616b23b6-9bb3-4465-9bb8-26c472f95f71" (UID: "616b23b6-9bb3-4465-9bb8-26c472f95f71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:15.993482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.993454    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:15.993565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.993486    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7zg\" (UniqueName: \"kubernetes.io/projected/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-api-access-7l7zg\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:15.993565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.993505    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616b23b6-9bb3-4465-9bb8-26c472f95f71-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:15.993565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.993521    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/616b23b6-9bb3-4465-9bb8-26c472f95f71-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:15.993565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:15.993537    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/616b23b6-9bb3-4465-9bb8-26c472f95f71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:16.761805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.761772    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679" event={"ID":"616b23b6-9bb3-4465-9bb8-26c472f95f71","Type":"ContainerDied","Data":"e071e03e99ec2c56d454a0ef7a48922cdc2ea6771c2ed1c4150a4d233081be56"}
Oct 30 19:00:16.761805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.761796    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"
Oct 30 19:00:16.762295 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.761836    4942 scope.go:122] "RemoveContainer" containerID="e2056f17fe4e3f7c1857fcc1c704f3a8c4ba0dd90fda44c1a4667a62df10b0be"
Oct 30 19:00:16.770980 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.770956    4942 scope.go:122] "RemoveContainer" containerID="cb0e06f91518f54d2bb03225836ce2684934017acd6c56c11de91eb6900469a8"
Oct 30 19:00:16.778156 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.778137    4942 scope.go:122] "RemoveContainer" containerID="08a2aca11f3a09aa1de3db6dc6f3198fd6d769008cc18473ea025e5b41ee019d"
Oct 30 19:00:16.787504 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.787483    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 19:00:16.795471 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:16.795443    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-predictive-lightgbm-v2-predictor-6bc49b69f-br679"]
Oct 30 19:00:17.545251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:17.543427    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" path="/var/lib/kubelet/pods/616b23b6-9bb3-4465-9bb8-26c472f95f71/volumes"
Oct 30 19:00:26.724191 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:00:26.724150    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:00:28.531371 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531337    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531673    4942 cpu_manager.go:414] "containerMap: removing container" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="storage-initializer"
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531683    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="storage-initializer"
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531695    4942 cpu_manager.go:414] "containerMap: removing container" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kserve-container"
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531700    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kserve-container"
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531707    4942 cpu_manager.go:414] "containerMap: removing container" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kube-rbac-proxy"
Oct 30 19:00:28.531732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531712    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kube-rbac-proxy"
Oct 30 19:00:28.531916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531771    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kserve-container"
Oct 30 19:00:28.531916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.531778    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="616b23b6-9bb3-4465-9bb8-26c472f95f71" containerName="kube-rbac-proxy"
Oct 30 19:00:28.536572 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.536555    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.543240 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.543215    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:00:28.543360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.543290    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 30 19:00:28.559378 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.559355    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:00:28.559485 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.559401    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 19:00:28.559485 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.559461    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:00:28.559986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.559970    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:00:28.566413 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.566392    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:00:28.644114 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.644081    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:00:28.702201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.702159    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9ds\" (UniqueName: \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.702335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.702229    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.702335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.702296    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.702335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.702331    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.702440 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.702404    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803045 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.802978    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803049    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803094    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803127    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9ds\" (UniqueName: \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803269 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803186    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803336 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:00:28.803320    4942 secret.go:189] Couldn't get secret e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Oct 30 19:00:28.803408 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:00:28.803397    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls podName:0f0cd3f0-61e6-4377-b279-f519a2fc6ef1 nodeName:}" failed. No retries permitted until 2026-10-30 19:00:29.303377232 +0000 UTC m=+1415.881472902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls") pod "isvc-sklearn-predictor-6479864f5b-7vs2p" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1") : secret "isvc-sklearn-predictor-serving-cert" not found
Oct 30 19:00:28.803522 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803504    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803729 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803711    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.803767 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.803740    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:28.816728 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:28.816703    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9ds\" (UniqueName: \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:29.307802 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.307760    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:29.310320 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.310296    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") pod \"isvc-sklearn-predictor-6479864f5b-7vs2p\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:29.453489 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.453448    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:29.735607 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.735579    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:00:29.738652 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:00:29.738619    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0cd3f0_61e6_4377_b279_f519a2fc6ef1.slice/crio-c9d7887c8856c2aa0d6f2a512d376e9ec0ac100ac7a0c3940776d09608c9d7b3 WatchSource:0}: Error finding container c9d7887c8856c2aa0d6f2a512d376e9ec0ac100ac7a0c3940776d09608c9d7b3: Status 404 returned error can't find the container with id c9d7887c8856c2aa0d6f2a512d376e9ec0ac100ac7a0c3940776d09608c9d7b3
Oct 30 19:00:29.817614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.817579    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerStarted","Data":"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"}
Oct 30 19:00:29.817715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:29.817681    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerStarted","Data":"c9d7887c8856c2aa0d6f2a512d376e9ec0ac100ac7a0c3940776d09608c9d7b3"}
Oct 30 19:00:33.837914 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:33.837882    4942 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerID="934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911" exitCode=0
Oct 30 19:00:33.838274 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:33.837937    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerDied","Data":"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"}
Oct 30 19:00:34.844211 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.844164    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerStarted","Data":"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"}
Oct 30 19:00:34.844709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.844251    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerStarted","Data":"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"}
Oct 30 19:00:34.844709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.844548    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:34.844709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.844604    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:34.846137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.846106    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:00:34.870935 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:34.870890    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podStartSLOduration=2.999803907 podStartE2EDuration="6.870878361s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871074454s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:34.869477401 +0000 UTC m=+1421.447573095" watchObservedRunningTime="2026-10-30 19:00:34.870878361 +0000 UTC m=+1421.448974051"
Oct 30 19:00:35.848810 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:35.848768    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:00:40.853973 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:40.853930    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:00:40.854595 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:40.854568    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:00:50.854501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:00:50.854459    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:00:56.725283 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:00:56.725244    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:01:00.855292 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:00.855254    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:10.854995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:10.854954    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:20.854773 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:20.854731    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:26.723758 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:01:26.723721    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:01:30.855487 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:30.855449    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:40.855439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:40.855344    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:49.537222 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:49.537168    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 19:01:53.667877 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:53.667835    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:01:53.669422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:53.669388    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:01:56.729166 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:01:56.729121    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:01:59.543204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:01:59.543165    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:02:09.278865 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:09.278828    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:11.313174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:11.313134    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:13.343427 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:13.343389    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:15.365509 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:15.365473    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:17.387589 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:17.387549    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:19.410137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:19.410106    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:21.443002 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:21.442958    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:23.467959 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:23.467928    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:25.490587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:25.490541    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:26.727876 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:02:26.727830    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:02:27.513643 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:27.513607    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:29.534493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:29.534460    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:31.556489 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:31.556459    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:33.578842 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:33.578815    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:35.600690 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:35.600655    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:37.623297 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:37.623264    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:39.655436 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:39.655403    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c83bbf5b66_isvc-sklearn-predictor-6479864f5b-7vs2p_0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/kserve-container/0.log"
Oct 30 19:02:39.947272 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:39.947231    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:02:39.947562 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:39.947527    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container" containerID="cri-o://2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e" gracePeriod=30
Oct 30 19:02:39.947662 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:39.947586    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kube-rbac-proxy" containerID="cri-o://90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757" gracePeriod=30
Oct 30 19:02:40.384240 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:40.384147    4942 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerID="90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757" exitCode=0
Oct 30 19:02:40.384240 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:40.384216    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerDied","Data":"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"}
Oct 30 19:02:40.849872 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:40.849824    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 19:02:45.710780 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.710753    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:02:45.826306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826202    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\" (UniqueName: \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\") pod \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") "
Oct 30 19:02:45.826306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826256    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") pod \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") "
Oct 30 19:02:45.826306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826299    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\") pod \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") "
Oct 30 19:02:45.826587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826348    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\") pod \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") "
Oct 30 19:02:45.826587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826425    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\") pod \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\" (UID: \"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1\") "
Oct 30 19:02:45.826750 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826714    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config" pod "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:45.826813 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826743    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location" pod "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:45.826813 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.826734    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert" pod "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:45.828463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.828435    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls" pod "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:45.828587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.828530    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds" pod "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" (UID: "0f0cd3f0-61e6-4377-b279-f519a2fc6ef1"). InnerVolumeSpecName "kube-api-access-tf9ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:45.927554 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.927517    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:45.927554 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.927549    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:45.927554 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.927560    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9ds\" (UniqueName: \"kubernetes.io/projected/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-api-access-tf9ds\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:45.927792 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.927571    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:45.927792 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:45.927580    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:46.414363 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.414318    4942 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerID="2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e" exitCode=0
Oct 30 19:02:46.414550 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.414382    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"
Oct 30 19:02:46.414550 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.414399    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerDied","Data":"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"}
Oct 30 19:02:46.414550 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.414460    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p" event={"ID":"0f0cd3f0-61e6-4377-b279-f519a2fc6ef1","Type":"ContainerDied","Data":"c9d7887c8856c2aa0d6f2a512d376e9ec0ac100ac7a0c3940776d09608c9d7b3"}
Oct 30 19:02:46.414550 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.414488    4942 scope.go:122] "RemoveContainer" containerID="90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"
Oct 30 19:02:46.423588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.423567    4942 scope.go:122] "RemoveContainer" containerID="2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"
Oct 30 19:02:46.431772 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.431750    4942 scope.go:122] "RemoveContainer" containerID="934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"
Oct 30 19:02:46.441319 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.441291    4942 scope.go:122] "RemoveContainer" containerID="90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"
Oct 30 19:02:46.441868 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:02:46.441846    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757\": container with ID starting with 90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757 not found: ID does not exist" containerID="90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"
Oct 30 19:02:46.441941 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.441879    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757"} err="failed to get container status \"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757\": rpc error: code = NotFound desc = could not find container \"90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757\": container with ID starting with 90c6ce0820f0dff413047649b6337b275c5e3a1d0f0f68764ddd6d2d0cde0757 not found: ID does not exist"
Oct 30 19:02:46.441941 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.441899    4942 scope.go:122] "RemoveContainer" containerID="2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"
Oct 30 19:02:46.442294 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:02:46.442277    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e\": container with ID starting with 2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e not found: ID does not exist" containerID="2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"
Oct 30 19:02:46.442351 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.442299    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e"} err="failed to get container status \"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e\": rpc error: code = NotFound desc = could not find container \"2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e\": container with ID starting with 2f05239447946b3f8c0a46cd70c222ebbd4487b12b6cf35dfa7c743c8b9ae77e not found: ID does not exist"
Oct 30 19:02:46.442351 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.442313    4942 scope.go:122] "RemoveContainer" containerID="934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"
Oct 30 19:02:46.442577 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:02:46.442560    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911\": container with ID starting with 934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911 not found: ID does not exist" containerID="934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"
Oct 30 19:02:46.442629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.442582    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911"} err="failed to get container status \"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911\": rpc error: code = NotFound desc = could not find container \"934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911\": container with ID starting with 934dca5ea885d6aa8d636e12aa94ae26a43396a094a44942fd56a505f8b84911 not found: ID does not exist"
Oct 30 19:02:46.445331 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.445303    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:02:46.459609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:46.459576    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-predictor-6479864f5b-7vs2p"]
Oct 30 19:02:47.543355 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.543317    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" path="/var/lib/kubelet/pods/0f0cd3f0-61e6-4377-b279-f519a2fc6ef1/volumes"
Oct 30 19:02:47.550863 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.550837    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:02:47.551165 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551153    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kube-rbac-proxy"
Oct 30 19:02:47.551204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551166    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kube-rbac-proxy"
Oct 30 19:02:47.551204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551195    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container"
Oct 30 19:02:47.551204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551201    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container"
Oct 30 19:02:47.551300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551212    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="storage-initializer"
Oct 30 19:02:47.551300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551217    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="storage-initializer"
Oct 30 19:02:47.551300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551277    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kserve-container"
Oct 30 19:02:47.551300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.551287    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0cd3f0-61e6-4377-b279-f519a2fc6ef1" containerName="kube-rbac-proxy"
Oct 30 19:02:47.556059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.556038    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.568463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.568441    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 30 19:02:47.568777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.568760    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:02:47.570846 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.570832    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:02:47.571585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.571558    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:02:47.630891 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.630858    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:02:47.630891 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.630888    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:02:47.631170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.630939    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:02:47.632268 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.632251    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:02:47.641368 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.641340    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.641450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.641375    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.641450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.641396    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.641560 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.641489    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.641613 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.641595    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzsd\" (UniqueName: \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.742546 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.742498    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzsd\" (UniqueName: \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.742765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.742557    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.742765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.742580    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.742765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.742599    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.742765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.742619    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.743044 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.743003    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.743328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.743305    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.743444 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.743404    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.745165 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.745149    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.760886 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.760859    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzsd\" (UniqueName: \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\") pod \"sklearn-v2-mlserver-predictor-677b8967-vktx7\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") " pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:47.873915 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:47.873825    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:02:48.157581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:48.157497    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:02:48.161024 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:02:48.160983    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac3fcf1_146b_449b_b12b_6f060b0a86d6.slice/crio-09948455c60caf33870f6dfd6531e9e2da482a1c4a4dd78d713b2019ea2458ce WatchSource:0}: Error finding container 09948455c60caf33870f6dfd6531e9e2da482a1c4a4dd78d713b2019ea2458ce: Status 404 returned error can't find the container with id 09948455c60caf33870f6dfd6531e9e2da482a1c4a4dd78d713b2019ea2458ce
Oct 30 19:02:48.433420 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:48.433378    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerStarted","Data":"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"}
Oct 30 19:02:48.433694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:48.433448    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerStarted","Data":"09948455c60caf33870f6dfd6531e9e2da482a1c4a4dd78d713b2019ea2458ce"}
Oct 30 19:02:52.452337 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:52.452239    4942 generic.go:356] "Generic (PLEG): container finished" podID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerID="5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a" exitCode=0
Oct 30 19:02:52.452337 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:52.452321    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerDied","Data":"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"}
Oct 30 19:02:52.453697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:02:52.453679    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:02:56.731507 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:02:56.731462    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:03:26.735443 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:03:26.735395    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:03:26.850591 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:26.850551    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:03:27.180317 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.180265    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:03:27.180501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.180403    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.189493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.189455    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"message-dumper-predictor-serving-cert\""
Oct 30 19:03:27.189632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.189510    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:03:27.189632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.189510    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Oct 30 19:03:27.189632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.189541    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:03:27.189632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.189539    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:03:27.201581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.201543    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.201779 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.201592    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.201779 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.201614    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzghx\" (UniqueName: \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.303047 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.302992    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.303252 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.303168    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.303252 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.303222    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzghx\" (UniqueName: \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.303799 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.303774    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.306119 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.306096    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.321738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.321702    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzghx\" (UniqueName: \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\") pod \"message-dumper-predictor-bbbfd7f9f-bt5gw\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") " pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.522246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.522156    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:27.833777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:27.833667    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:03:27.836851 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:03:27.836805    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda036d50c_7f3d_4117_862c_757d809042db.slice/crio-c032cee162218825589fbd4f37d1ce706edbbdc4813ee19001e430b5674f5433 WatchSource:0}: Error finding container c032cee162218825589fbd4f37d1ce706edbbdc4813ee19001e430b5674f5433: Status 404 returned error can't find the container with id c032cee162218825589fbd4f37d1ce706edbbdc4813ee19001e430b5674f5433
Oct 30 19:03:28.646211 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:28.646164    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerStarted","Data":"c032cee162218825589fbd4f37d1ce706edbbdc4813ee19001e430b5674f5433"}
Oct 30 19:03:30.656937 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:30.656902    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerStarted","Data":"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"}
Oct 30 19:03:30.657408 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:30.656970    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerStarted","Data":"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"}
Oct 30 19:03:30.657408 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:30.657069    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:30.686511 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:30.686449    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" podStartSLOduration=2.763121242 podStartE2EDuration="4.686432978s" totalImagesPullingTime="1.923311736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:30.685211797 +0000 UTC m=+1597.263307489" watchObservedRunningTime="2026-10-30 19:03:30.686432978 +0000 UTC m=+1597.264528671"
Oct 30 19:03:31.662422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:31.662383    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:31.664131 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:31.664109    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:38.678278 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:38.678235    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:03:46.907631 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.907587    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:03:46.911469 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.911436    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:46.917439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.917407    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:03:46.917658 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.917563    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-logger-predictor-serving-cert\""
Oct 30 19:03:46.918404 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.918375    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 30 19:03:46.918571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.918413    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:03:46.929805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.929764    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:03:46.993953 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.993911    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:46.994189 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.994019    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:46.994189 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.994114    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:46.994189 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.994151    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khb5\" (UniqueName: \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:46.994379 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:46.994188    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095247    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095308    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095441    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khb5\" (UniqueName: \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095518    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095595    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.095908 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.095873    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.096182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.096156    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.096422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.096395    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.099300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.099266    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.124949 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.124903    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khb5\" (UniqueName: \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\") pod \"isvc-logger-predictor-54d79f78f6-jgx6n\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") " pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.235970 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.235703    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:47.543775 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:03:47.543728    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ebe615_0b8d_4ea0_b312_c71836e6225e.slice/crio-cc9986836eeee481ddedb96c456fc144f42c734aee893776a5f4a549b9c7aa01 WatchSource:0}: Error finding container cc9986836eeee481ddedb96c456fc144f42c734aee893776a5f4a549b9c7aa01: Status 404 returned error can't find the container with id cc9986836eeee481ddedb96c456fc144f42c734aee893776a5f4a549b9c7aa01
Oct 30 19:03:47.545082 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.545050    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:03:47.743451 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.743401    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerStarted","Data":"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"}
Oct 30 19:03:47.743651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:47.743542    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerStarted","Data":"cc9986836eeee481ddedb96c456fc144f42c734aee893776a5f4a549b9c7aa01"}
Oct 30 19:03:52.772531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:52.772484    4942 generic.go:356] "Generic (PLEG): container finished" podID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerID="2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044" exitCode=0
Oct 30 19:03:52.773057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:52.772544    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerDied","Data":"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"}
Oct 30 19:03:53.781284 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781233    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerStarted","Data":"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"}
Oct 30 19:03:53.781840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781325    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerStarted","Data":"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"}
Oct 30 19:03:53.781840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781357    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerStarted","Data":"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"}
Oct 30 19:03:53.781840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781707    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:53.781840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781773    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:53.781840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.781809    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:53.784235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.784186    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:03:53.785229 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.785190    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:03:53.810461 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:53.810372    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podStartSLOduration=3.564383706 podStartE2EDuration="7.810347576s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.24596387s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:53.806069686 +0000 UTC m=+1620.384165380" watchObservedRunningTime="2026-10-30 19:03:53.810347576 +0000 UTC m=+1620.388443269"
Oct 30 19:03:54.787366 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:54.787313    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:03:54.787856 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:54.787643    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:03:56.734162 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:03:56.734118    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:03:59.793097 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:59.793044    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:03:59.793695 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:59.793665    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:03:59.794127 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:03:59.794096    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:09.794094 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:09.794046    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:09.794585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:09.794508    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:19.794404 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:19.794355    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:19.794939 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:19.794835    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:26.728508 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:04:26.728464    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:04:29.794022 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:29.793968    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:29.794525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:29.794498    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:39.794064 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:39.793945    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:39.794581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:39.794394    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:49.794491 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:49.794450    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:49.795151 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:49.795020    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:52.077433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:52.077396    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerStarted","Data":"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"}
Oct 30 19:04:52.077899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:52.077472    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:04:52.077899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:52.077623    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:04:52.077899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:52.077671    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerStarted","Data":"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"}
Oct 30 19:04:52.107220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:52.107182    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podStartSLOduration=2.006636866 podStartE2EDuration="2m5.107171354s" totalImagesPullingTime="1m59.121976558s" totalInitContainerRuntime="3.97855793s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:52.105130466 +0000 UTC m=+1678.683226162" watchObservedRunningTime="2026-10-30 19:04:52.107171354 +0000 UTC m=+1678.685267040"
Oct 30 19:04:56.724602 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:04:56.724564    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:04:58.087936 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:58.087900    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:04:59.793716 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:59.793682    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:04:59.794125 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:04:59.794038    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:05:02.536666 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:02.536615    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:05:02.537178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:02.536890    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:05:12.536650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:12.536603    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:05:12.537115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:12.536980    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:05:22.538060 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:22.537984    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:05:22.538453 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:22.538195    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:05:26.723774 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:26.723738    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:05:28.145160 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:28.145120    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:05:32.485682 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.485638    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-3deaac58e4_message-dumper-predictor-bbbfd7f9f-bt5gw_a036d50c-7f3d-4117-862c-757d809042db/kserve-container/0.log"
Oct 30 19:05:32.785494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.785419    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:05:32.785717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.785694    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" containerID="cri-o://0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f" gracePeriod=30
Oct 30 19:05:32.785831 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.785795    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" containerID="cri-o://dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa" gracePeriod=30
Oct 30 19:05:32.785926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.785817    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" containerID="cri-o://2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640" gracePeriod=30
Oct 30 19:05:32.840310 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.840268    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:05:32.840529 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.840505    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kserve-container" containerID="cri-o://7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e" gracePeriod=30
Oct 30 19:05:32.840820 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:32.840801    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kube-rbac-proxy" containerID="cri-o://eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90" gracePeriod=30
Oct 30 19:05:33.092807 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.092782    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:05:33.238464 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.238433    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\" (UniqueName: \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\") pod \"a036d50c-7f3d-4117-862c-757d809042db\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") "
Oct 30 19:05:33.238723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.238504    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\") pod \"a036d50c-7f3d-4117-862c-757d809042db\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") "
Oct 30 19:05:33.238723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.238562    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\") pod \"a036d50c-7f3d-4117-862c-757d809042db\" (UID: \"a036d50c-7f3d-4117-862c-757d809042db\") "
Oct 30 19:05:33.238915 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.238891    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config" pod "a036d50c-7f3d-4117-862c-757d809042db" (UID: "a036d50c-7f3d-4117-862c-757d809042db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:33.240607 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.240579    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls" pod "a036d50c-7f3d-4117-862c-757d809042db" (UID: "a036d50c-7f3d-4117-862c-757d809042db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:33.240709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.240629    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx" pod "a036d50c-7f3d-4117-862c-757d809042db" (UID: "a036d50c-7f3d-4117-862c-757d809042db"). InnerVolumeSpecName "kube-api-access-xzghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:33.254141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254114    4942 generic.go:356] "Generic (PLEG): container finished" podID="a036d50c-7f3d-4117-862c-757d809042db" containerID="eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90" exitCode=0
Oct 30 19:05:33.254141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254138    4942 generic.go:356] "Generic (PLEG): container finished" podID="a036d50c-7f3d-4117-862c-757d809042db" containerID="7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e" exitCode=2
Oct 30 19:05:33.254303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254163    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"
Oct 30 19:05:33.254303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254190    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerDied","Data":"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"}
Oct 30 19:05:33.254303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254245    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerDied","Data":"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"}
Oct 30 19:05:33.254303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254274    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw" event={"ID":"a036d50c-7f3d-4117-862c-757d809042db","Type":"ContainerDied","Data":"c032cee162218825589fbd4f37d1ce706edbbdc4813ee19001e430b5674f5433"}
Oct 30 19:05:33.254474 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.254312    4942 scope.go:122] "RemoveContainer" containerID="eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"
Oct 30 19:05:33.256948 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.256928    4942 generic.go:356] "Generic (PLEG): container finished" podID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerID="2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640" exitCode=0
Oct 30 19:05:33.257075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.256988    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerDied","Data":"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"}
Oct 30 19:05:33.263732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.263717    4942 scope.go:122] "RemoveContainer" containerID="7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"
Oct 30 19:05:33.271181 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271165    4942 scope.go:122] "RemoveContainer" containerID="eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"
Oct 30 19:05:33.271441 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:33.271425    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90\": container with ID starting with eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90 not found: ID does not exist" containerID="eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"
Oct 30 19:05:33.271482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271451    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"} err="failed to get container status \"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90\": rpc error: code = NotFound desc = could not find container \"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90\": container with ID starting with eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90 not found: ID does not exist"
Oct 30 19:05:33.271482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271465    4942 scope.go:122] "RemoveContainer" containerID="7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"
Oct 30 19:05:33.271680 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:33.271660    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e\": container with ID starting with 7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e not found: ID does not exist" containerID="7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"
Oct 30 19:05:33.271717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271686    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"} err="failed to get container status \"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e\": rpc error: code = NotFound desc = could not find container \"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e\": container with ID starting with 7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e not found: ID does not exist"
Oct 30 19:05:33.271717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271700    4942 scope.go:122] "RemoveContainer" containerID="eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"
Oct 30 19:05:33.271920 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271901    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90"} err="failed to get container status \"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90\": rpc error: code = NotFound desc = could not find container \"eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90\": container with ID starting with eea597407a4bfd7b354d89aaeceef549f2d923c8ee94cf4c5048c760f3c70f90 not found: ID does not exist"
Oct 30 19:05:33.271967 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.271921    4942 scope.go:122] "RemoveContainer" containerID="7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"
Oct 30 19:05:33.272132 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.272114    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e"} err="failed to get container status \"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e\": rpc error: code = NotFound desc = could not find container \"7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e\": container with ID starting with 7375b22bc63d5bd9aa582781f162b94ce1a36b911499e20a5825be8fa605d55e not found: ID does not exist"
Oct 30 19:05:33.325700 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.325635    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:05:33.339490 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.339465    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a036d50c-7f3d-4117-862c-757d809042db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:33.339490 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.339488    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzghx\" (UniqueName: \"kubernetes.io/projected/a036d50c-7f3d-4117-862c-757d809042db-kube-api-access-xzghx\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:33.339610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.339499    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a036d50c-7f3d-4117-862c-757d809042db-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:33.346330 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.346309    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/message-dumper-predictor-bbbfd7f9f-bt5gw"]
Oct 30 19:05:33.541584 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:33.541554    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a036d50c-7f3d-4117-862c-757d809042db" path="/var/lib/kubelet/pods/a036d50c-7f3d-4117-862c-757d809042db/volumes"
Oct 30 19:05:34.787926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:34.787886    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:05:38.090530 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:38.090499    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:05:38.284763 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:38.284728    4942 generic.go:356] "Generic (PLEG): container finished" podID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerID="0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f" exitCode=0
Oct 30 19:05:38.284938 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:38.284806    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerDied","Data":"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"}
Oct 30 19:05:39.787582 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:39.787544    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:05:42.537484 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:42.537433    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:05:42.537925 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:42.537796    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:05:44.787431 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:44.787383    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:05:44.787878 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:44.787577    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:05:48.004140 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.004105    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:05:48.004615 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.004343    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container" containerID="cri-o://054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e" gracePeriod=30
Oct 30 19:05:48.004615 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.004374    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy" containerID="cri-o://2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51" gracePeriod=30
Oct 30 19:05:48.082765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.082728    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 30 19:05:48.329202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.329112    4942 generic.go:356] "Generic (PLEG): container finished" podID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerID="2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51" exitCode=0
Oct 30 19:05:48.329202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:48.329186    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerDied","Data":"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"}
Oct 30 19:05:49.130182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:49.130132    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.49:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:49.787916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:49.787868    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:05:52.536662 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:52.536612    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:05:52.537137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:52.536934    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:05:53.082808 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:53.082767    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 30 19:05:54.648972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.648944    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:05:54.711296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711222    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\") pod \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") "
Oct 30 19:05:54.711296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711272    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\") pod \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") "
Oct 30 19:05:54.711478 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711319    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\" (UniqueName: \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\") pod \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") "
Oct 30 19:05:54.711478 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711350    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\") pod \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") "
Oct 30 19:05:54.711478 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711378    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\") pod \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\" (UID: \"7ac3fcf1-146b-449b-b12b-6f060b0a86d6\") "
Oct 30 19:05:54.711703 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711675    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location" pod "7ac3fcf1-146b-449b-b12b-6f060b0a86d6" (UID: "7ac3fcf1-146b-449b-b12b-6f060b0a86d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:54.711761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711727    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert" pod "7ac3fcf1-146b-449b-b12b-6f060b0a86d6" (UID: "7ac3fcf1-146b-449b-b12b-6f060b0a86d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:54.711761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.711741    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config" pod "7ac3fcf1-146b-449b-b12b-6f060b0a86d6" (UID: "7ac3fcf1-146b-449b-b12b-6f060b0a86d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:54.713435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.713408    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls" pod "7ac3fcf1-146b-449b-b12b-6f060b0a86d6" (UID: "7ac3fcf1-146b-449b-b12b-6f060b0a86d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:54.713435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.713423    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd" pod "7ac3fcf1-146b-449b-b12b-6f060b0a86d6" (UID: "7ac3fcf1-146b-449b-b12b-6f060b0a86d6"). InnerVolumeSpecName "kube-api-access-ngzsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:54.787450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.787420    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:05:54.812486 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.812463    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:54.812486 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.812484    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:54.812610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.812496    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzsd\" (UniqueName: \"kubernetes.io/projected/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-api-access-ngzsd\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:54.812610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.812505    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:54.812610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:54.812514    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac3fcf1-146b-449b-b12b-6f060b0a86d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:55.359841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.359807    4942 generic.go:356] "Generic (PLEG): container finished" podID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerID="054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e" exitCode=0
Oct 30 19:05:55.359988 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.359868    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"
Oct 30 19:05:55.359988 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.359887    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerDied","Data":"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"}
Oct 30 19:05:55.359988 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.359956    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7" event={"ID":"7ac3fcf1-146b-449b-b12b-6f060b0a86d6","Type":"ContainerDied","Data":"09948455c60caf33870f6dfd6531e9e2da482a1c4a4dd78d713b2019ea2458ce"}
Oct 30 19:05:55.360127 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.359995    4942 scope.go:122] "RemoveContainer" containerID="2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"
Oct 30 19:05:55.369301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.369286    4942 scope.go:122] "RemoveContainer" containerID="054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"
Oct 30 19:05:55.376788 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.376767    4942 scope.go:122] "RemoveContainer" containerID="5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"
Oct 30 19:05:55.383721 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.383702    4942 scope.go:122] "RemoveContainer" containerID="2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"
Oct 30 19:05:55.383988 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:55.383967    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51\": container with ID starting with 2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51 not found: ID does not exist" containerID="2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"
Oct 30 19:05:55.384064 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.383997    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51"} err="failed to get container status \"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51\": rpc error: code = NotFound desc = could not find container \"2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51\": container with ID starting with 2a08d8d36fbe8b2a09ef7fe9deeda8f6c386848861fd3fab694dec3f9416be51 not found: ID does not exist"
Oct 30 19:05:55.384064 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.384035    4942 scope.go:122] "RemoveContainer" containerID="054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"
Oct 30 19:05:55.384265 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:55.384247    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e\": container with ID starting with 054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e not found: ID does not exist" containerID="054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"
Oct 30 19:05:55.384355 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.384278    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e"} err="failed to get container status \"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e\": rpc error: code = NotFound desc = could not find container \"054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e\": container with ID starting with 054d66decbac05e54a03eade9eb2c9f6a618b93a51e2b44e19626a433e0c401e not found: ID does not exist"
Oct 30 19:05:55.384355 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.384293    4942 scope.go:122] "RemoveContainer" containerID="5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"
Oct 30 19:05:55.384561 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:55.384544    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a\": container with ID starting with 5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a not found: ID does not exist" containerID="5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"
Oct 30 19:05:55.384607 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.384565    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a"} err="failed to get container status \"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a\": rpc error: code = NotFound desc = could not find container \"5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a\": container with ID starting with 5fb69a7935164cc390f232340495a956d9d55b0f010146fd3d70b7c5f75a1a7a not found: ID does not exist"
Oct 30 19:05:55.388236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.388216    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:05:55.402357 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.402337    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/sklearn-v2-mlserver-predictor-677b8967-vktx7"]
Oct 30 19:05:55.541819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:55.541796    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" path="/var/lib/kubelet/pods/7ac3fcf1-146b-449b-b12b-6f060b0a86d6/volumes"
Oct 30 19:05:56.724688 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:05:56.724639    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:05:57.621413 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621381    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:05:57.621749 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621737    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kube-rbac-proxy"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621751    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kube-rbac-proxy"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621761    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621766    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621776    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kserve-container"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621781    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kserve-container"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621789    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="storage-initializer"
Oct 30 19:05:57.621793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621794    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="storage-initializer"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621803    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621808    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621861    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kserve-container"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621868    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kserve-container"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621875    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac3fcf1-146b-449b-b12b-6f060b0a86d6" containerName="kube-rbac-proxy"
Oct 30 19:05:57.622251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.621885    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="a036d50c-7f3d-4117-862c-757d809042db" containerName="kube-rbac-proxy"
Oct 30 19:05:57.626472 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.626451    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.632363 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.632336    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:05:57.632363 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.632352    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 30 19:05:57.632556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.632368    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:05:57.632556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.632378    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:05:57.632556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.632395    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:05:57.634542 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.634521    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:05:57.637116 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.637096    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:05:57.732108 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.732080    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:05:57.733312 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.733291    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.733386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.733327    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.733386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.733354    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.733449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.733394    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.733449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.733432    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r686q\" (UniqueName: \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834177 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834143    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834186    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834224    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834248    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834279    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r686q\" (UniqueName: \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834599    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.834959 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834927    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.835062 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.834933    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.836761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.836736    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.847464 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.847443    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r686q\" (UniqueName: \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\") pod \"isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:57.945400 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:57.945369    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:05:58.235610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:58.235587    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:05:58.238112 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:05:58.238082    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd8eebd_8ca6_404a_881d_519ea9af800e.slice/crio-70120aceae2f438b447e28c28b25ea10c0eef0417688af3603b792957043abf3 WatchSource:0}: Error finding container 70120aceae2f438b447e28c28b25ea10c0eef0417688af3603b792957043abf3: Status 404 returned error can't find the container with id 70120aceae2f438b447e28c28b25ea10c0eef0417688af3603b792957043abf3
Oct 30 19:05:58.374889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:58.374844    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerStarted","Data":"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"}
Oct 30 19:05:58.375068 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:58.374923    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerStarted","Data":"70120aceae2f438b447e28c28b25ea10c0eef0417688af3603b792957043abf3"}
Oct 30 19:05:59.788097 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:05:59.788059    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 19:06:02.537588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.537543    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 19:06:02.538081 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.537685    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:06:02.538149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.538102    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:02.538245 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.538229    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:06:02.937753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.937725    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:06:02.978422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.978359    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\") pod \"53ebe615-0b8d-4ea0-b312-c71836e6225e\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") "
Oct 30 19:06:02.978577 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.978468    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\") pod \"53ebe615-0b8d-4ea0-b312-c71836e6225e\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") "
Oct 30 19:06:02.978577 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.978531    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\" (UniqueName: \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\") pod \"53ebe615-0b8d-4ea0-b312-c71836e6225e\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") "
Oct 30 19:06:02.978806 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.978744    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location" pod "53ebe615-0b8d-4ea0-b312-c71836e6225e" (UID: "53ebe615-0b8d-4ea0-b312-c71836e6225e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:02.979612 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.979586    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\") pod \"53ebe615-0b8d-4ea0-b312-c71836e6225e\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") "
Oct 30 19:06:02.979755 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.979639    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\") pod \"53ebe615-0b8d-4ea0-b312-c71836e6225e\" (UID: \"53ebe615-0b8d-4ea0-b312-c71836e6225e\") "
Oct 30 19:06:02.981195 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.979972    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ebe615-0b8d-4ea0-b312-c71836e6225e-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:02.981195 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.980045    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert" pod "53ebe615-0b8d-4ea0-b312-c71836e6225e" (UID: "53ebe615-0b8d-4ea0-b312-c71836e6225e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:02.981195 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.980266    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config" pod "53ebe615-0b8d-4ea0-b312-c71836e6225e" (UID: "53ebe615-0b8d-4ea0-b312-c71836e6225e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:02.981435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.981410    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls" pod "53ebe615-0b8d-4ea0-b312-c71836e6225e" (UID: "53ebe615-0b8d-4ea0-b312-c71836e6225e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:02.981784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:02.981755    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5" pod "53ebe615-0b8d-4ea0-b312-c71836e6225e" (UID: "53ebe615-0b8d-4ea0-b312-c71836e6225e"). InnerVolumeSpecName "kube-api-access-8khb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:03.080752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.080691    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:03.080752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.080719    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:03.080752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.080730    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ebe615-0b8d-4ea0-b312-c71836e6225e-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:03.080752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.080740    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khb5\" (UniqueName: \"kubernetes.io/projected/53ebe615-0b8d-4ea0-b312-c71836e6225e-kube-api-access-8khb5\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:03.398417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.398336    4942 generic.go:356] "Generic (PLEG): container finished" podID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerID="dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa" exitCode=0
Oct 30 19:06:03.398564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.398427    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"
Oct 30 19:06:03.398564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.398419    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerDied","Data":"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"}
Oct 30 19:06:03.398636 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.398610    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n" event={"ID":"53ebe615-0b8d-4ea0-b312-c71836e6225e","Type":"ContainerDied","Data":"cc9986836eeee481ddedb96c456fc144f42c734aee893776a5f4a549b9c7aa01"}
Oct 30 19:06:03.398666 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.398657    4942 scope.go:122] "RemoveContainer" containerID="dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"
Oct 30 19:06:03.408626 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.408608    4942 scope.go:122] "RemoveContainer" containerID="2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"
Oct 30 19:06:03.416128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.416111    4942 scope.go:122] "RemoveContainer" containerID="0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"
Oct 30 19:06:03.423577 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.423555    4942 scope.go:122] "RemoveContainer" containerID="2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"
Oct 30 19:06:03.424133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.424116    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:06:03.429570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.429549    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-logger-predictor-54d79f78f6-jgx6n"]
Oct 30 19:06:03.431231 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.431218    4942 scope.go:122] "RemoveContainer" containerID="dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"
Oct 30 19:06:03.431501 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:03.431483    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa\": container with ID starting with dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa not found: ID does not exist" containerID="dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"
Oct 30 19:06:03.431558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.431510    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa"} err="failed to get container status \"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa\": rpc error: code = NotFound desc = could not find container \"dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa\": container with ID starting with dae4f1dc2b1217efc69b7053ad70aa24d84682008b63bf28965824e6093d1cfa not found: ID does not exist"
Oct 30 19:06:03.431558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.431525    4942 scope.go:122] "RemoveContainer" containerID="2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"
Oct 30 19:06:03.431786 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:03.431768    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640\": container with ID starting with 2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640 not found: ID does not exist" containerID="2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"
Oct 30 19:06:03.431858 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.431796    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640"} err="failed to get container status \"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640\": rpc error: code = NotFound desc = could not find container \"2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640\": container with ID starting with 2b61e6edb28d90b808895451591014cc5aae97c9bdba90a8e98cbde19edcc640 not found: ID does not exist"
Oct 30 19:06:03.431858 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.431817    4942 scope.go:122] "RemoveContainer" containerID="0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"
Oct 30 19:06:03.432062 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:03.432045    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f\": container with ID starting with 0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f not found: ID does not exist" containerID="0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"
Oct 30 19:06:03.432110 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.432067    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f"} err="failed to get container status \"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f\": rpc error: code = NotFound desc = could not find container \"0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f\": container with ID starting with 0812602723bb96107fcd2219d40618e9ae2bd177d9fb79ca99fbb72de634da6f not found: ID does not exist"
Oct 30 19:06:03.432110 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.432080    4942 scope.go:122] "RemoveContainer" containerID="2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"
Oct 30 19:06:03.432313 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:03.432291    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044\": container with ID starting with 2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044 not found: ID does not exist" containerID="2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"
Oct 30 19:06:03.432356 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.432316    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044"} err="failed to get container status \"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044\": rpc error: code = NotFound desc = could not find container \"2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044\": container with ID starting with 2e4f8bc3821db22f215c309d4119885cf6ef64198d0eaacb2eaae33bd4a60044 not found: ID does not exist"
Oct 30 19:06:03.542212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:03.542175    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" path="/var/lib/kubelet/pods/53ebe615-0b8d-4ea0-b312-c71836e6225e/volumes"
Oct 30 19:06:06.413027 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:06.412966    4942 generic.go:356] "Generic (PLEG): container finished" podID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerID="62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5" exitCode=0
Oct 30 19:06:06.413460 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:06.413039    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerDied","Data":"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"}
Oct 30 19:06:07.420256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:07.420217    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerStarted","Data":"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"}
Oct 30 19:06:07.420866 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:07.420282    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerStarted","Data":"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"}
Oct 30 19:06:07.420866 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:07.420489    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:07.448486 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:07.448426    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podStartSLOduration=2.877483532 podStartE2EDuration="10.448410643s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.570927111s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:07.446453119 +0000 UTC m=+1754.024548812" watchObservedRunningTime="2026-10-30 19:06:07.448410643 +0000 UTC m=+1754.026506339"
Oct 30 19:06:08.425502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:08.425464    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:08.426986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:08.426945    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 19:06:09.430192 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:09.430155    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 19:06:14.434812 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:14.434778    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:14.435321 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:14.435296    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 19:06:24.436544 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:24.436503    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:26.726807 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:26.726758    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:06:31.714135 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:31.714098    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:06:31.714850 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:31.714819    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container" containerID="cri-o://948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6" gracePeriod=30
Oct 30 19:06:31.715221 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:31.715187    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kube-rbac-proxy" containerID="cri-o://689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db" gracePeriod=30
Oct 30 19:06:32.551031 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:32.550975    4942 generic.go:356] "Generic (PLEG): container finished" podID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerID="689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db" exitCode=0
Oct 30 19:06:32.551208 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:32.551053    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerDied","Data":"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"}
Oct 30 19:06:34.430784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:34.430740    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 19:06:37.567532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.567489    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:37.575073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.574968    4942 generic.go:356] "Generic (PLEG): container finished" podID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerID="948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6" exitCode=139
Oct 30 19:06:37.575073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.575058    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"
Oct 30 19:06:37.575073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.575057    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerDied","Data":"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"}
Oct 30 19:06:37.575300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.575137    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb" event={"ID":"ccd8eebd-8ca6-404a-881d-519ea9af800e","Type":"ContainerDied","Data":"70120aceae2f438b447e28c28b25ea10c0eef0417688af3603b792957043abf3"}
Oct 30 19:06:37.575300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.575174    4942 scope.go:122] "RemoveContainer" containerID="689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"
Oct 30 19:06:37.582088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.582053    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\") pod \"ccd8eebd-8ca6-404a-881d-519ea9af800e\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") "
Oct 30 19:06:37.582277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.582103    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\") pod \"ccd8eebd-8ca6-404a-881d-519ea9af800e\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") "
Oct 30 19:06:37.582277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.582142    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\") pod \"ccd8eebd-8ca6-404a-881d-519ea9af800e\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") "
Oct 30 19:06:37.582277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.582191    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\" (UniqueName: \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\") pod \"ccd8eebd-8ca6-404a-881d-519ea9af800e\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") "
Oct 30 19:06:37.582801 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.582765    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config" pod "ccd8eebd-8ca6-404a-881d-519ea9af800e" (UID: "ccd8eebd-8ca6-404a-881d-519ea9af800e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:37.584341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.584308    4942 scope.go:122] "RemoveContainer" containerID="948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"
Oct 30 19:06:37.585148 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.585062    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls" pod "ccd8eebd-8ca6-404a-881d-519ea9af800e" (UID: "ccd8eebd-8ca6-404a-881d-519ea9af800e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:37.585268 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.585183    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q" pod "ccd8eebd-8ca6-404a-881d-519ea9af800e" (UID: "ccd8eebd-8ca6-404a-881d-519ea9af800e"). InnerVolumeSpecName "kube-api-access-r686q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:37.598548 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.598515    4942 scope.go:122] "RemoveContainer" containerID="62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"
Oct 30 19:06:37.609288 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.609249    4942 scope.go:122] "RemoveContainer" containerID="689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"
Oct 30 19:06:37.609709 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:37.609682    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db\": container with ID starting with 689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db not found: ID does not exist" containerID="689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"
Oct 30 19:06:37.609823 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.609721    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db"} err="failed to get container status \"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db\": rpc error: code = NotFound desc = could not find container \"689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db\": container with ID starting with 689745021d6137726e80d58c2efa7f6684fbd58842fb84324e7f9f852ef297db not found: ID does not exist"
Oct 30 19:06:37.609823 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.609742    4942 scope.go:122] "RemoveContainer" containerID="948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"
Oct 30 19:06:37.610142 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:37.610115    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6\": container with ID starting with 948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6 not found: ID does not exist" containerID="948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"
Oct 30 19:06:37.610208 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.610152    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6"} err="failed to get container status \"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6\": rpc error: code = NotFound desc = could not find container \"948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6\": container with ID starting with 948a7d0ce0584f51b91035bafc6df233e61af73bb41d9091e9c35a3c8d4f1fc6 not found: ID does not exist"
Oct 30 19:06:37.610208 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.610178    4942 scope.go:122] "RemoveContainer" containerID="62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"
Oct 30 19:06:37.610509 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:37.610485    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5\": container with ID starting with 62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5 not found: ID does not exist" containerID="62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"
Oct 30 19:06:37.610566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.610517    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5"} err="failed to get container status \"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5\": rpc error: code = NotFound desc = could not find container \"62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5\": container with ID starting with 62406e7854692a7cfbf3d97f606cb71ea054f471f3aa0f0f6688e2ceef0b15f5 not found: ID does not exist"
Oct 30 19:06:37.618330 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.618285    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location" pod "ccd8eebd-8ca6-404a-881d-519ea9af800e" (UID: "ccd8eebd-8ca6-404a-881d-519ea9af800e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:37.683360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683310    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\") pod \"ccd8eebd-8ca6-404a-881d-519ea9af800e\" (UID: \"ccd8eebd-8ca6-404a-881d-519ea9af800e\") "
Oct 30 19:06:37.683597 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683572    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccd8eebd-8ca6-404a-881d-519ea9af800e-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:37.683674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683600    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccd8eebd-8ca6-404a-881d-519ea9af800e-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:37.683674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683616    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:37.683674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683631    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r686q\" (UniqueName: \"kubernetes.io/projected/ccd8eebd-8ca6-404a-881d-519ea9af800e-kube-api-access-r686q\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:37.683779 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.683670    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert" pod "ccd8eebd-8ca6-404a-881d-519ea9af800e" (UID: "ccd8eebd-8ca6-404a-881d-519ea9af800e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:37.784756 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.784707    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ccd8eebd-8ca6-404a-881d-519ea9af800e-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:37.900699 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.900655    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:06:37.905144 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:37.905110    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-runtime-predictor-5c9689c6bb-6wzdb"]
Oct 30 19:06:39.272857 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.272812    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273246    4942 cpu_manager.go:414] "containerMap: removing container" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="storage-initializer"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273260    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="storage-initializer"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273271    4942 cpu_manager.go:414] "containerMap: removing container" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273277    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273287    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273295    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container"
Oct 30 19:06:39.273308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273309    4942 cpu_manager.go:414] "containerMap: removing container" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273314    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273326    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="storage-initializer"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273332    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="storage-initializer"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273340    4942 cpu_manager.go:414] "containerMap: removing container" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273344    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273351    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273355    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273415    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kserve-container"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273424    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kserve-container"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273430    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273436    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ebe615-0b8d-4ea0-b312-c71836e6225e" containerName="agent"
Oct 30 19:06:39.273527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.273443    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" containerName="kube-rbac-proxy"
Oct 30 19:06:39.278393 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.278366    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.284525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.284490    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:06:39.284525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.284500    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 30 19:06:39.284741 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.284498    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:06:39.286145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.286117    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:06:39.286145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.286118    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:06:39.286145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.286119    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:06:39.286349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.286122    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:06:39.292737 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.292683    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:06:39.296560 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.296521    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.296719 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.296569    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.296719 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.296650    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.296719 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.296668    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ncp\" (UniqueName: \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.296841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.296720    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.397950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.397895    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.397957    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398045    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398096    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398125    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ncp\" (UniqueName: \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398726 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398689    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.398947 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398918    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.399079 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.398950    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.401355 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.401316    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.414767 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.414730    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ncp\" (UniqueName: \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\") pod \"isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.543211 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.543122    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd8eebd-8ca6-404a-881d-519ea9af800e" path="/var/lib/kubelet/pods/ccd8eebd-8ca6-404a-881d-519ea9af800e/volumes"
Oct 30 19:06:39.597885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.597830    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:39.887153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:39.887060    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:06:39.894586 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:06:39.894514    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d0806a_279e_4bc2_a58d_a6c4216c0b70.slice/crio-eee19fead298b04d1ea73e3621d223b2109da9634bd45d8a991e61633495918d WatchSource:0}: Error finding container eee19fead298b04d1ea73e3621d223b2109da9634bd45d8a991e61633495918d: Status 404 returned error can't find the container with id eee19fead298b04d1ea73e3621d223b2109da9634bd45d8a991e61633495918d
Oct 30 19:06:40.594115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:40.594064    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerStarted","Data":"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"}
Oct 30 19:06:40.594542 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:40.594150    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerStarted","Data":"eee19fead298b04d1ea73e3621d223b2109da9634bd45d8a991e61633495918d"}
Oct 30 19:06:44.615699 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:44.615588    4942 generic.go:356] "Generic (PLEG): container finished" podID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerID="dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf" exitCode=0
Oct 30 19:06:44.615699 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:44.615678    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerDied","Data":"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"}
Oct 30 19:06:45.623082 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:45.623041    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerStarted","Data":"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"}
Oct 30 19:06:45.623526 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:45.623111    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerStarted","Data":"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"}
Oct 30 19:06:45.623526 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:45.623315    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:45.654055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:45.653977    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podStartSLOduration=2.383226825 podStartE2EDuration="6.653960514s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.270733689s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:45.651618099 +0000 UTC m=+1792.229713791" watchObservedRunningTime="2026-10-30 19:06:45.653960514 +0000 UTC m=+1792.232056294"
Oct 30 19:06:46.627447 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:46.627406    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:52.637356 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:52.637323    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:06:53.696849 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:53.696807    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:06:53.698629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:06:53.698601    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:06:56.727916 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:06:56.727873    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:07:22.743661 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:22.743612    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:07:26.726591 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:07:26.726545    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:07:32.640951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:32.640915    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:07:39.604020 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:39.603924    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:07:39.604575 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:39.604196    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kserve-container" containerID="cri-o://3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708" gracePeriod=30
Oct 30 19:07:39.604575 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:39.604237    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy" containerID="cri-o://8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1" gracePeriod=30
Oct 30 19:07:39.874558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:39.874477    4942 generic.go:356] "Generic (PLEG): container finished" podID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerID="8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1" exitCode=0
Oct 30 19:07:39.874558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:39.874538    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerDied","Data":"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"}
Oct 30 19:07:42.631959 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:42.631913    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 30 19:07:47.650056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.650029    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:07:47.685350 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685316    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\") pod \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") "
Oct 30 19:07:47.685531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685363    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\") pod \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") "
Oct 30 19:07:47.685531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685401    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\" (UniqueName: \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\") pod \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") "
Oct 30 19:07:47.685531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685445    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\") pod \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") "
Oct 30 19:07:47.685531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685514    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\") pod \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\" (UID: \"c7d0806a-279e-4bc2-a58d-a6c4216c0b70\") "
Oct 30 19:07:47.685844 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.685811    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert" pod "c7d0806a-279e-4bc2-a58d-a6c4216c0b70" (UID: "c7d0806a-279e-4bc2-a58d-a6c4216c0b70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:47.686227 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.686050    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config" pod "c7d0806a-279e-4bc2-a58d-a6c4216c0b70" (UID: "c7d0806a-279e-4bc2-a58d-a6c4216c0b70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:47.686227 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.686056    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location" pod "c7d0806a-279e-4bc2-a58d-a6c4216c0b70" (UID: "c7d0806a-279e-4bc2-a58d-a6c4216c0b70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:47.687642 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.687608    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls" pod "c7d0806a-279e-4bc2-a58d-a6c4216c0b70" (UID: "c7d0806a-279e-4bc2-a58d-a6c4216c0b70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:47.687738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.687716    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp" pod "c7d0806a-279e-4bc2-a58d-a6c4216c0b70" (UID: "c7d0806a-279e-4bc2-a58d-a6c4216c0b70"). InnerVolumeSpecName "kube-api-access-k2ncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:47.786660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.786565    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.786660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.786597    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.786660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.786608    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ncp\" (UniqueName: \"kubernetes.io/projected/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-api-access-k2ncp\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.786660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.786619    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.786660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.786631    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d0806a-279e-4bc2-a58d-a6c4216c0b70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.911388 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.911352    4942 generic.go:356] "Generic (PLEG): container finished" podID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerID="3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708" exitCode=0
Oct 30 19:07:47.911551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.911393    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerDied","Data":"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"}
Oct 30 19:07:47.911551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.911407    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"
Oct 30 19:07:47.911551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.911464    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" event={"ID":"c7d0806a-279e-4bc2-a58d-a6c4216c0b70","Type":"ContainerDied","Data":"eee19fead298b04d1ea73e3621d223b2109da9634bd45d8a991e61633495918d"}
Oct 30 19:07:47.911551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.911501    4942 scope.go:122] "RemoveContainer" containerID="8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"
Oct 30 19:07:47.920451 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.920428    4942 scope.go:122] "RemoveContainer" containerID="3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"
Oct 30 19:07:47.928095 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.928075    4942 scope.go:122] "RemoveContainer" containerID="dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"
Oct 30 19:07:47.935955 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.935922    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:07:47.936039 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936024    4942 scope.go:122] "RemoveContainer" containerID="8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"
Oct 30 19:07:47.936313 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:07:47.936296    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1\": container with ID starting with 8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1 not found: ID does not exist" containerID="8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"
Oct 30 19:07:47.936366 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936321    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1"} err="failed to get container status \"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1\": rpc error: code = NotFound desc = could not find container \"8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1\": container with ID starting with 8028eba0d4d96ab715de495c04f3c35f2e9e80d2738269cf328888d61d3835a1 not found: ID does not exist"
Oct 30 19:07:47.936366 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936351    4942 scope.go:122] "RemoveContainer" containerID="3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"
Oct 30 19:07:47.936580 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:07:47.936561    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708\": container with ID starting with 3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708 not found: ID does not exist" containerID="3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"
Oct 30 19:07:47.936621 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936584    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708"} err="failed to get container status \"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708\": rpc error: code = NotFound desc = could not find container \"3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708\": container with ID starting with 3606273da046fd60305314996692c831a115cfe0bf4a1621142c2fd7dfb27708 not found: ID does not exist"
Oct 30 19:07:47.936621 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936599    4942 scope.go:122] "RemoveContainer" containerID="dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"
Oct 30 19:07:47.936817 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:07:47.936800    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf\": container with ID starting with dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf not found: ID does not exist" containerID="dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"
Oct 30 19:07:47.936861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.936821    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf"} err="failed to get container status \"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf\": rpc error: code = NotFound desc = could not find container \"dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf\": container with ID starting with dd9f8b9be552c6ba5c240ea9605a58ebb531efec70f50fed022ee416d12004bf not found: ID does not exist"
Oct 30 19:07:47.943665 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:47.943642    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt"]
Oct 30 19:07:48.632410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:48.632349    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-runtime-predictor-b84cd8bc-f4vrt" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": context deadline exceeded"
Oct 30 19:07:49.087838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.087801    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:07:49.089212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089182    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="storage-initializer"
Oct 30 19:07:49.089365 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089351    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="storage-initializer"
Oct 30 19:07:49.089491 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089480    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kserve-container"
Oct 30 19:07:49.089569 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089559    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kserve-container"
Oct 30 19:07:49.089652 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089642    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy"
Oct 30 19:07:49.089781 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.089770    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy"
Oct 30 19:07:49.090122 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.090106    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kserve-container"
Oct 30 19:07:49.090236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.090225    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" containerName="kube-rbac-proxy"
Oct 30 19:07:49.096234 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.096214    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.106946 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.106921    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:07:49.107088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.106932    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:07:49.107088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.106932    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:07:49.107088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.106978    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:07:49.107088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.107061    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 30 19:07:49.107266 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.107088    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:07:49.107266 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.107259    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:07:49.108243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.108222    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:07:49.200669 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.200629    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.200669 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.200678    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.200876 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.200720    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.200876 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.200802    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.200876 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.200837    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7pq\" (UniqueName: \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.301587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.301547    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.301587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.301593    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.301848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.301614    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7pq\" (UniqueName: \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.301848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.301672    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.301848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.301706    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.302202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.302177    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.302421 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.302401    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.302457 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.302404    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.304080 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.304061    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.315120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.315099    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7pq\" (UniqueName: \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\") pod \"isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.414345 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.414260    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:49.541747 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.541706    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d0806a-279e-4bc2-a58d-a6c4216c0b70" path="/var/lib/kubelet/pods/c7d0806a-279e-4bc2-a58d-a6c4216c0b70/volumes"
Oct 30 19:07:49.699975 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.699941    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:07:49.703182 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:07:49.703144    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54daa28d_1675_491b_9a91_f694e5f6da9d.slice/crio-b6ab089c9f3235d66540a0bf2b7dbbcf5ed9bb81d958fa772dfe8e8774d1e4ad WatchSource:0}: Error finding container b6ab089c9f3235d66540a0bf2b7dbbcf5ed9bb81d958fa772dfe8e8774d1e4ad: Status 404 returned error can't find the container with id b6ab089c9f3235d66540a0bf2b7dbbcf5ed9bb81d958fa772dfe8e8774d1e4ad
Oct 30 19:07:49.923113 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.923077    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerStarted","Data":"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"}
Oct 30 19:07:49.923315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:49.923145    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerStarted","Data":"b6ab089c9f3235d66540a0bf2b7dbbcf5ed9bb81d958fa772dfe8e8774d1e4ad"}
Oct 30 19:07:53.942139 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:53.942098    4942 generic.go:356] "Generic (PLEG): container finished" podID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerID="6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a" exitCode=0
Oct 30 19:07:53.942553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:53.942151    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerDied","Data":"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"}
Oct 30 19:07:54.949512 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:54.949468    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerStarted","Data":"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"}
Oct 30 19:07:54.949922 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:54.949581    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerStarted","Data":"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"}
Oct 30 19:07:54.949922 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:54.949794    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:54.983046 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:54.982978    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podStartSLOduration=1.966480648 podStartE2EDuration="5.982964251s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.016483603s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:54.982256091 +0000 UTC m=+1861.560351785" watchObservedRunningTime="2026-10-30 19:07:54.982964251 +0000 UTC m=+1861.561060210"
Oct 30 19:07:55.954143 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:55.954110    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:07:55.955554 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:55.955527    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:07:56.723914 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:07:56.723877    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:07:56.957935 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:07:56.957898    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:01.963117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:01.963084    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:08:01.963717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:01.963690    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:11.964243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:11.964201    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:21.964026 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:21.963979    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:26.724473 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:08:26.724436    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:08:31.964630 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:31.964583    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:41.964549 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:41.964498    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:51.963672 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:51.963632    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:08:54.869025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.868979    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:08:54.872817 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.872796    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:54.888995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.888966    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:08:54.888995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.888965    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:08:54.889235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.889217    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:08:54.890600 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.890579    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 30 19:08:54.891500 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.891478    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:08:54.891883 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.891861    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:08:54.891996 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.891910    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:08:54.893130 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.893106    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:08:54.942275 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.942236    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:54.942446 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.942291    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:54.942446 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.942315    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:54.942568 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.942439    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:54.942568 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:54.942487    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj57h\" (UniqueName: \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.043350 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.043313    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.043532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.043363    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj57h\" (UniqueName: \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.043532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.043399    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.043532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.043446    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.043532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.043476    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.044436 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.044082    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.044573 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.044496    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.044808 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.044779    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.053868 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.053836    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.056840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.056810    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj57h\" (UniqueName: \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\") pod \"isvc-autogluon-v2-predictor-854b885cc8-52qmh\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.190294 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.190243    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:08:55.474841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:55.474805    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:08:55.477833 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:08:55.477801    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafba245_1bc1_44b9_b17b_4f68642c7fec.slice/crio-4415a8e951d81e0af3582340e5097398310159fc96aab4832735084797fb1e07 WatchSource:0}: Error finding container 4415a8e951d81e0af3582340e5097398310159fc96aab4832735084797fb1e07: Status 404 returned error can't find the container with id 4415a8e951d81e0af3582340e5097398310159fc96aab4832735084797fb1e07
Oct 30 19:08:56.213359 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:56.213314    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerStarted","Data":"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"}
Oct 30 19:08:56.213874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:08:56.213391    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerStarted","Data":"4415a8e951d81e0af3582340e5097398310159fc96aab4832735084797fb1e07"}
Oct 30 19:08:56.725733 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:08:56.725695    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:09:01.963653 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:01.963613    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:09:02.241951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:02.241861    4942 generic.go:356] "Generic (PLEG): container finished" podID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerID="5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d" exitCode=0
Oct 30 19:09:02.241951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:02.241938    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerDied","Data":"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"}
Oct 30 19:09:02.243301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:02.243282    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:09:11.964350 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:11.963900    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:09:16.537415 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:16.537371    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 19:09:26.538198 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:26.538166    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:09:26.728675 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:09:26.728640    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:09:30.434326 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:30.434289    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:09:30.434713 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:30.434586    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container" containerID="cri-o://271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047" gracePeriod=30
Oct 30 19:09:30.434713 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:30.434633    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kube-rbac-proxy" containerID="cri-o://2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5" gracePeriod=30
Oct 30 19:09:31.386927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:31.386891    4942 generic.go:356] "Generic (PLEG): container finished" podID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerID="2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5" exitCode=0
Oct 30 19:09:31.387135 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:31.386969    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerDied","Data":"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"}
Oct 30 19:09:31.958746 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:31.958702    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 30 19:09:35.931489 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.931466    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:09:35.999186 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999157    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\") pod \"54daa28d-1675-491b-9a91-f694e5f6da9d\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") "
Oct 30 19:09:35.999334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999206    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\" (UniqueName: \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\") pod \"54daa28d-1675-491b-9a91-f694e5f6da9d\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") "
Oct 30 19:09:35.999334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999249    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\") pod \"54daa28d-1675-491b-9a91-f694e5f6da9d\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") "
Oct 30 19:09:35.999334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999281    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\") pod \"54daa28d-1675-491b-9a91-f694e5f6da9d\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") "
Oct 30 19:09:35.999510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999335    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\") pod \"54daa28d-1675-491b-9a91-f694e5f6da9d\" (UID: \"54daa28d-1675-491b-9a91-f694e5f6da9d\") "
Oct 30 19:09:35.999714 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999686    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert" pod "54daa28d-1675-491b-9a91-f694e5f6da9d" (UID: "54daa28d-1675-491b-9a91-f694e5f6da9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:35.999714 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999693    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location" pod "54daa28d-1675-491b-9a91-f694e5f6da9d" (UID: "54daa28d-1675-491b-9a91-f694e5f6da9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.999858 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:35.999722    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config" pod "54daa28d-1675-491b-9a91-f694e5f6da9d" (UID: "54daa28d-1675-491b-9a91-f694e5f6da9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:36.002191 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.002167    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq" pod "54daa28d-1675-491b-9a91-f694e5f6da9d" (UID: "54daa28d-1675-491b-9a91-f694e5f6da9d"). InnerVolumeSpecName "kube-api-access-gq7pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:36.002291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.002252    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls" pod "54daa28d-1675-491b-9a91-f694e5f6da9d" (UID: "54daa28d-1675-491b-9a91-f694e5f6da9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:36.100120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.100069    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7pq\" (UniqueName: \"kubernetes.io/projected/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-api-access-gq7pq\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.100244 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.100122    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.100244 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.100139    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54daa28d-1675-491b-9a91-f694e5f6da9d-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.100244 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.100157    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54daa28d-1675-491b-9a91-f694e5f6da9d-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.100244 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.100172    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54daa28d-1675-491b-9a91-f694e5f6da9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.414796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.414718    4942 generic.go:356] "Generic (PLEG): container finished" podID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerID="271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047" exitCode=0
Oct 30 19:09:36.414796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.414789    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"
Oct 30 19:09:36.414984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.414777    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerDied","Data":"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"}
Oct 30 19:09:36.414984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.414953    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5" event={"ID":"54daa28d-1675-491b-9a91-f694e5f6da9d","Type":"ContainerDied","Data":"b6ab089c9f3235d66540a0bf2b7dbbcf5ed9bb81d958fa772dfe8e8774d1e4ad"}
Oct 30 19:09:36.415092 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.414996    4942 scope.go:122] "RemoveContainer" containerID="2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"
Oct 30 19:09:36.428428 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.428342    4942 scope.go:122] "RemoveContainer" containerID="271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"
Oct 30 19:09:36.436956 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.436939    4942 scope.go:122] "RemoveContainer" containerID="6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"
Oct 30 19:09:36.441656 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.441631    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:09:36.446397 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.446370    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-predictor-7d6685c6cb-jvkd5"]
Oct 30 19:09:36.448313 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.448297    4942 scope.go:122] "RemoveContainer" containerID="2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"
Oct 30 19:09:36.448745 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:09:36.448721    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5\": container with ID starting with 2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5 not found: ID does not exist" containerID="2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"
Oct 30 19:09:36.448830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.448756    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5"} err="failed to get container status \"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5\": rpc error: code = NotFound desc = could not find container \"2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5\": container with ID starting with 2e2b7589cb1813d3b62ca3186db784a0726acdcd1cff7c69bdc124807625f3f5 not found: ID does not exist"
Oct 30 19:09:36.448830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.448778    4942 scope.go:122] "RemoveContainer" containerID="271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"
Oct 30 19:09:36.449118 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:09:36.449100    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047\": container with ID starting with 271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047 not found: ID does not exist" containerID="271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"
Oct 30 19:09:36.449213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.449126    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047"} err="failed to get container status \"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047\": rpc error: code = NotFound desc = could not find container \"271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047\": container with ID starting with 271fe65fd9d9a3d15c75caee851dbc75950d83f173761f896c377db22bdcc047 not found: ID does not exist"
Oct 30 19:09:36.449213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.449146    4942 scope.go:122] "RemoveContainer" containerID="6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"
Oct 30 19:09:36.449431 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:09:36.449405    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a\": container with ID starting with 6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a not found: ID does not exist" containerID="6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"
Oct 30 19:09:36.449529 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:36.449436    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a"} err="failed to get container status \"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a\": rpc error: code = NotFound desc = could not find container \"6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a\": container with ID starting with 6c84d4bc371973d49f104c05835e482cf56d8533ea05b1e2c98c4ce87ecb457a not found: ID does not exist"
Oct 30 19:09:37.542788 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.542758    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" path="/var/lib/kubelet/pods/54daa28d-1675-491b-9a91-f694e5f6da9d/volumes"
Oct 30 19:09:37.917883 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.917785    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:09:37.918299 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918279    4942 cpu_manager.go:414] "containerMap: removing container" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container"
Oct 30 19:09:37.918406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918300    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container"
Oct 30 19:09:37.918406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918321    4942 cpu_manager.go:414] "containerMap: removing container" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="storage-initializer"
Oct 30 19:09:37.918406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918329    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="storage-initializer"
Oct 30 19:09:37.918406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918353    4942 cpu_manager.go:414] "containerMap: removing container" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kube-rbac-proxy"
Oct 30 19:09:37.918406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918361    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kube-rbac-proxy"
Oct 30 19:09:37.918659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918464    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kube-rbac-proxy"
Oct 30 19:09:37.918659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.918482    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="54daa28d-1675-491b-9a91-f694e5f6da9d" containerName="kserve-container"
Oct 30 19:09:37.958080 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.957952    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:09:37.958080 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.958054    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:37.963300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.963275    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:09:37.963449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.963339    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:09:37.963449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.963337    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 30 19:09:37.963449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.963439    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:09:37.964920 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.964896    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:09:37.966714 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.966694    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 30 19:09:37.970142 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:37.970123    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:09:38.018658 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.018634    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2phs\" (UniqueName: \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.018775 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.018669    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.018775 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.018697    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.018906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.018838    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.018906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.018860    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120098    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120264 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120146    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120264 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120222    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2phs\" (UniqueName: \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120393 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120295    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120393 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120351    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.120650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.120626    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.121117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.121090    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.121117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.121104    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.122838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.122816    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.134513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.134491    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2phs\" (UniqueName: \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\") pod \"isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.333242 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.333202    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:38.636739 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:38.635478    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:09:38.639190 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:09:38.639160    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5385cce7_a667_4d32_86df_e0a9d5b469c8.slice/crio-7ee45f91cd955f6fad32c7077857993da2eb84c4b77108c07a271dcd9b9b162d WatchSource:0}: Error finding container 7ee45f91cd955f6fad32c7077857993da2eb84c4b77108c07a271dcd9b9b162d: Status 404 returned error can't find the container with id 7ee45f91cd955f6fad32c7077857993da2eb84c4b77108c07a271dcd9b9b162d
Oct 30 19:09:39.434505 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:39.434411    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerStarted","Data":"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"}
Oct 30 19:09:39.434703 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:39.434523    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerStarted","Data":"7ee45f91cd955f6fad32c7077857993da2eb84c4b77108c07a271dcd9b9b162d"}
Oct 30 19:09:43.458724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:43.458675    4942 generic.go:356] "Generic (PLEG): container finished" podID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerID="74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74" exitCode=0
Oct 30 19:09:43.459180 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:43.458749    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerDied","Data":"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"}
Oct 30 19:09:44.466231 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.466192    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerStarted","Data":"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"}
Oct 30 19:09:44.466687 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.466359    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerStarted","Data":"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"}
Oct 30 19:09:44.466844 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.466819    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:44.466931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.466903    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:44.468394 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.468359    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:09:44.492553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:44.492470    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podStartSLOduration=2.910025505 podStartE2EDuration="7.492458563s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.582433058s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:44.49179243 +0000 UTC m=+1971.069888133" watchObservedRunningTime="2026-10-30 19:09:44.492458563 +0000 UTC m=+1971.070554250"
Oct 30 19:09:45.472314 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:45.472270    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:09:50.477338 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:50.477279    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:09:50.478055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:09:50.477907    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:09:56.730307 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:09:56.730270    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:10:00.477928 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:00.477886    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:10.478630 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:10.478583    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:20.478220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:20.478182    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:26.798450 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:10:26.798408    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:10:30.478277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:30.478227    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:40.478655 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:40.478553    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:48.777961 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:48.777922    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerStarted","Data":"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"}
Oct 30 19:10:48.778539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:48.777997    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerStarted","Data":"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"}
Oct 30 19:10:48.778539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:48.778060    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:10:48.778539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:48.778199    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:10:48.805507 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:48.805447    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" podStartSLOduration=3.003365547 podStartE2EDuration="1m54.805430539s" totalImagesPullingTime="1m45.781950381s" totalInitContainerRuntime="6.020114611s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:48.804225887 +0000 UTC m=+2035.382321579" watchObservedRunningTime="2026-10-30 19:10:48.805430539 +0000 UTC m=+2035.383526227"
Oct 30 19:10:50.478664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:50.478622    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:10:54.791276 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:10:54.791244    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:10:56.728219 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:10:56.728168    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:11:00.478668 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:00.478622    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:11:10.478328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:10.478285    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:11:19.542629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:19.542589    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:11:26.727604 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:11:26.727560    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:11:29.252588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.252535    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:11:29.253147 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.252821    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" containerID="cri-o://a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb" gracePeriod=30
Oct 30 19:11:29.253147 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.252860    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kube-rbac-proxy" containerID="cri-o://91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6" gracePeriod=30
Oct 30 19:11:29.537104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.536975    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 19:11:29.974428 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.974387    4942 generic.go:356] "Generic (PLEG): container finished" podID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerID="91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6" exitCode=0
Oct 30 19:11:29.974637 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:29.974464    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerDied","Data":"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"}
Oct 30 19:11:30.472912 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:30.472861    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Oct 30 19:11:34.808296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.808265    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:11:34.867669 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.867637    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\") pod \"5385cce7-a667-4d32-86df-e0a9d5b469c8\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") "
Oct 30 19:11:34.867879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.867683    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\") pod \"5385cce7-a667-4d32-86df-e0a9d5b469c8\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") "
Oct 30 19:11:34.867879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.867745    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\" (UniqueName: \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\") pod \"5385cce7-a667-4d32-86df-e0a9d5b469c8\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") "
Oct 30 19:11:34.867879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.867813    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\") pod \"5385cce7-a667-4d32-86df-e0a9d5b469c8\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") "
Oct 30 19:11:34.867879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.867843    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\") pod \"5385cce7-a667-4d32-86df-e0a9d5b469c8\" (UID: \"5385cce7-a667-4d32-86df-e0a9d5b469c8\") "
Oct 30 19:11:34.868217 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.868129    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config" pod "5385cce7-a667-4d32-86df-e0a9d5b469c8" (UID: "5385cce7-a667-4d32-86df-e0a9d5b469c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:34.868217 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.868195    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert" pod "5385cce7-a667-4d32-86df-e0a9d5b469c8" (UID: "5385cce7-a667-4d32-86df-e0a9d5b469c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:34.868368 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.868232    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location" pod "5385cce7-a667-4d32-86df-e0a9d5b469c8" (UID: "5385cce7-a667-4d32-86df-e0a9d5b469c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:34.870437 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.870408    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls" pod "5385cce7-a667-4d32-86df-e0a9d5b469c8" (UID: "5385cce7-a667-4d32-86df-e0a9d5b469c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:34.870526 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.870447    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs" pod "5385cce7-a667-4d32-86df-e0a9d5b469c8" (UID: "5385cce7-a667-4d32-86df-e0a9d5b469c8"). InnerVolumeSpecName "kube-api-access-j2phs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:34.968784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.968751    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.968784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.968779    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5385cce7-a667-4d32-86df-e0a9d5b469c8-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.968784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.968790    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2phs\" (UniqueName: \"kubernetes.io/projected/5385cce7-a667-4d32-86df-e0a9d5b469c8-kube-api-access-j2phs\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.969055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.968799    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5385cce7-a667-4d32-86df-e0a9d5b469c8-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.969055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.968813    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5385cce7-a667-4d32-86df-e0a9d5b469c8-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.996861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.996828    4942 generic.go:356] "Generic (PLEG): container finished" podID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerID="a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb" exitCode=0
Oct 30 19:11:34.997023 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.996887    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"
Oct 30 19:11:34.997023 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.996908    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerDied","Data":"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"}
Oct 30 19:11:34.997023 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.996970    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx" event={"ID":"5385cce7-a667-4d32-86df-e0a9d5b469c8","Type":"ContainerDied","Data":"7ee45f91cd955f6fad32c7077857993da2eb84c4b77108c07a271dcd9b9b162d"}
Oct 30 19:11:34.997023 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:34.996999    4942 scope.go:122] "RemoveContainer" containerID="91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"
Oct 30 19:11:35.006429 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.006399    4942 scope.go:122] "RemoveContainer" containerID="a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"
Oct 30 19:11:35.014623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.014605    4942 scope.go:122] "RemoveContainer" containerID="74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"
Oct 30 19:11:35.022265 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.022244    4942 scope.go:122] "RemoveContainer" containerID="91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"
Oct 30 19:11:35.022665 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:11:35.022638    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6\": container with ID starting with 91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6 not found: ID does not exist" containerID="91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"
Oct 30 19:11:35.022738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.022674    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6"} err="failed to get container status \"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6\": rpc error: code = NotFound desc = could not find container \"91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6\": container with ID starting with 91ffcb00c4d6b4152eef5930992a7218a08be1d4c8356ca183117f5a2395cfd6 not found: ID does not exist"
Oct 30 19:11:35.022738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.022697    4942 scope.go:122] "RemoveContainer" containerID="a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"
Oct 30 19:11:35.022995 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:11:35.022968    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb\": container with ID starting with a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb not found: ID does not exist" containerID="a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"
Oct 30 19:11:35.023076 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.023021    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb"} err="failed to get container status \"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb\": rpc error: code = NotFound desc = could not find container \"a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb\": container with ID starting with a50d1b27732b07c8c9af2e8747a4885f02683ffc0f75bf24245e2d188a35ebfb not found: ID does not exist"
Oct 30 19:11:35.023076 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.023043    4942 scope.go:122] "RemoveContainer" containerID="74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"
Oct 30 19:11:35.023435 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:11:35.023367    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74\": container with ID starting with 74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74 not found: ID does not exist" containerID="74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"
Oct 30 19:11:35.023435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.023407    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74"} err="failed to get container status \"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74\": rpc error: code = NotFound desc = could not find container \"74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74\": container with ID starting with 74bd5b3c06ed93894acfd5e99db19f2815554cdcaf526c35e6139ec59f5d9b74 not found: ID does not exist"
Oct 30 19:11:35.024556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.024536    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:11:35.029777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.029756    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-v2-mixed-predictor-769d89b677-kpzgx"]
Oct 30 19:11:35.541706 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:35.541668    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" path="/var/lib/kubelet/pods/5385cce7-a667-4d32-86df-e0a9d5b469c8/volumes"
Oct 30 19:11:53.725897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:53.725859    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:11:53.728590 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:11:53.728568    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:11:56.724500 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:11:56.724456    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:12:26.724612 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:12:26.724569    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:12:27.569482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:27.569445    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:12:38.579224 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:38.579177    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:41.654094 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:41.654052    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:12:41.654502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:41.654377    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kserve-container" containerID="cri-o://ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae" gracePeriod=30
Oct 30 19:12:41.654502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:41.654448    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kube-rbac-proxy" containerID="cri-o://fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a" gracePeriod=30
Oct 30 19:12:42.282476 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:42.282442    4942 generic.go:356] "Generic (PLEG): container finished" podID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerID="fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a" exitCode=0
Oct 30 19:12:42.282656 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:42.282513    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerDied","Data":"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"}
Oct 30 19:12:43.103984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.103949    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:12:43.245711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.245623    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\") pod \"eafba245-1bc1-44b9-b17b-4f68642c7fec\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") "
Oct 30 19:12:43.245711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.245668    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\") pod \"eafba245-1bc1-44b9-b17b-4f68642c7fec\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") "
Oct 30 19:12:43.245711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.245708    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\") pod \"eafba245-1bc1-44b9-b17b-4f68642c7fec\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") "
Oct 30 19:12:43.246037 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.245730    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\") pod \"eafba245-1bc1-44b9-b17b-4f68642c7fec\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") "
Oct 30 19:12:43.246037 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.245785    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\" (UniqueName: \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\") pod \"eafba245-1bc1-44b9-b17b-4f68642c7fec\" (UID: \"eafba245-1bc1-44b9-b17b-4f68642c7fec\") "
Oct 30 19:12:43.246158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.246121    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config" pod "eafba245-1bc1-44b9-b17b-4f68642c7fec" (UID: "eafba245-1bc1-44b9-b17b-4f68642c7fec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:43.246158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.246133    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert" pod "eafba245-1bc1-44b9-b17b-4f68642c7fec" (UID: "eafba245-1bc1-44b9-b17b-4f68642c7fec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:43.247984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.247963    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls" pod "eafba245-1bc1-44b9-b17b-4f68642c7fec" (UID: "eafba245-1bc1-44b9-b17b-4f68642c7fec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:43.248071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.247985    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h" pod "eafba245-1bc1-44b9-b17b-4f68642c7fec" (UID: "eafba245-1bc1-44b9-b17b-4f68642c7fec"). InnerVolumeSpecName "kube-api-access-dj57h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:43.249365 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.249342    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location" pod "eafba245-1bc1-44b9-b17b-4f68642c7fec" (UID: "eafba245-1bc1-44b9-b17b-4f68642c7fec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:43.290088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.290056    4942 generic.go:356] "Generic (PLEG): container finished" podID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerID="ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae" exitCode=0
Oct 30 19:12:43.290261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.290117    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"
Oct 30 19:12:43.290261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.290152    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerDied","Data":"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"}
Oct 30 19:12:43.290261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.290212    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh" event={"ID":"eafba245-1bc1-44b9-b17b-4f68642c7fec","Type":"ContainerDied","Data":"4415a8e951d81e0af3582340e5097398310159fc96aab4832735084797fb1e07"}
Oct 30 19:12:43.290261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.290239    4942 scope.go:122] "RemoveContainer" containerID="fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"
Oct 30 19:12:43.298871 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.298851    4942 scope.go:122] "RemoveContainer" containerID="ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"
Oct 30 19:12:43.306279 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.306260    4942 scope.go:122] "RemoveContainer" containerID="5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"
Oct 30 19:12:43.313761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.313744    4942 scope.go:122] "RemoveContainer" containerID="fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"
Oct 30 19:12:43.314041 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:12:43.314024    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a\": container with ID starting with fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a not found: ID does not exist" containerID="fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"
Oct 30 19:12:43.314093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.314050    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a"} err="failed to get container status \"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a\": rpc error: code = NotFound desc = could not find container \"fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a\": container with ID starting with fb8f9e76dc7ead6f9341f76f4b52f8cc44b9e34f9adfc4edc517c195db06758a not found: ID does not exist"
Oct 30 19:12:43.314093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.314068    4942 scope.go:122] "RemoveContainer" containerID="ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"
Oct 30 19:12:43.314295 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:12:43.314277    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae\": container with ID starting with ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae not found: ID does not exist" containerID="ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"
Oct 30 19:12:43.314334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.314299    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae"} err="failed to get container status \"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae\": rpc error: code = NotFound desc = could not find container \"ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae\": container with ID starting with ae6235fcd28e28bbd5bdd0472913026408ef46abe51e05e6c547bc4015c647ae not found: ID does not exist"
Oct 30 19:12:43.314334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.314312    4942 scope.go:122] "RemoveContainer" containerID="5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"
Oct 30 19:12:43.314528 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:12:43.314504    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d\": container with ID starting with 5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d not found: ID does not exist" containerID="5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"
Oct 30 19:12:43.314566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.314533    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d"} err="failed to get container status \"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d\": rpc error: code = NotFound desc = could not find container \"5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d\": container with ID starting with 5c4c069947c607f5cb013f31b051d5000f3a0c4398341d2b0f158537c6d5df9d not found: ID does not exist"
Oct 30 19:12:43.318381 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.318359    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:12:43.332721 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.332701    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-predictor-854b885cc8-52qmh"]
Oct 30 19:12:43.346684 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.346664    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:43.346769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.346686    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafba245-1bc1-44b9-b17b-4f68642c7fec-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:43.346769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.346699    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafba245-1bc1-44b9-b17b-4f68642c7fec-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:43.346769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.346708    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafba245-1bc1-44b9-b17b-4f68642c7fec-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:43.346769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.346717    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj57h\" (UniqueName: \"kubernetes.io/projected/eafba245-1bc1-44b9-b17b-4f68642c7fec-kube-api-access-dj57h\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:43.543283 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:43.543207    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" path="/var/lib/kubelet/pods/eafba245-1bc1-44b9-b17b-4f68642c7fec/volumes"
Oct 30 19:12:45.172267 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172227    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172587    4942 cpu_manager.go:414] "containerMap: removing container" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kserve-container"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172599    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kserve-container"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172612    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172617    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172628    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kube-rbac-proxy"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172633    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kube-rbac-proxy"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172642    4942 cpu_manager.go:414] "containerMap: removing container" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="storage-initializer"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172647    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="storage-initializer"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172654    4942 cpu_manager.go:414] "containerMap: removing container" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kube-rbac-proxy"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172659    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kube-rbac-proxy"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172680    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="storage-initializer"
Oct 30 19:12:45.172705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172685    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="storage-initializer"
Oct 30 19:12:45.173088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172737    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kserve-container"
Oct 30 19:12:45.173088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172746    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kserve-container"
Oct 30 19:12:45.173088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172753    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="5385cce7-a667-4d32-86df-e0a9d5b469c8" containerName="kube-rbac-proxy"
Oct 30 19:12:45.173088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.172759    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafba245-1bc1-44b9-b17b-4f68642c7fec" containerName="kube-rbac-proxy"
Oct 30 19:12:45.177598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.177572    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.190236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190203    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:12:45.190539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190521    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:12:45.190651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190376    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:12:45.190750 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190530    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:12:45.190813 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190762    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:12:45.190813 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190665    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Oct 30 19:12:45.190913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.190866    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Oct 30 19:12:45.192376 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.192346    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:12:45.263485 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.263438    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.263485 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.263488    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.263754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.263589    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.263754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.263663    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.263754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.263692    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94qd\" (UniqueName: \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.364540 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.364493    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.364755 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.364547    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.364755 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.364590    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.364755 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.364631    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.364922 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.364753    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94qd\" (UniqueName: \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.365118 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.365094    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.365378 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.365358    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.365428 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.365360    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.367198 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.367175    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.379213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.379182    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94qd\" (UniqueName: \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\") pod \"isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.501034 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.500970    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:12:45.784997 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:45.784921    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:12:45.793273 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:12:45.793219    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6db8ee_819b_4719_ab25_ac0f2fa59b79.slice/crio-868e1347048c7d43f391293c380b35ff68fa337ba2244a28becc78e793a30191 WatchSource:0}: Error finding container 868e1347048c7d43f391293c380b35ff68fa337ba2244a28becc78e793a30191: Status 404 returned error can't find the container with id 868e1347048c7d43f391293c380b35ff68fa337ba2244a28becc78e793a30191
Oct 30 19:12:46.305370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:46.305326    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerStarted","Data":"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"}
Oct 30 19:12:46.305879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:12:46.305392    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerStarted","Data":"868e1347048c7d43f391293c380b35ff68fa337ba2244a28becc78e793a30191"}
Oct 30 19:12:56.729751 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:12:56.729696    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:13:18.445149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:18.445112    4942 generic.go:356] "Generic (PLEG): container finished" podID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerID="70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761" exitCode=0
Oct 30 19:13:18.445638 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:18.445192    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerDied","Data":"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"}
Oct 30 19:13:19.452224 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:19.452188    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerStarted","Data":"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"}
Oct 30 19:13:19.452627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:19.452255    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerStarted","Data":"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"}
Oct 30 19:13:19.452627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:19.452580    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:13:19.452708 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:19.452637    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:13:19.479938 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:19.479876    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" podStartSLOduration=2.068812988 podStartE2EDuration="34.479859208s" totalImagesPullingTime="0s" totalInitContainerRuntime="32.41104622s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:19.47868943 +0000 UTC m=+2186.056785121" watchObservedRunningTime="2026-10-30 19:13:19.479859208 +0000 UTC m=+2186.057954896"
Oct 30 19:13:25.462251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:13:25.462219    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:13:26.724866 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:13:26.724828    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:13:56.723950 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:13:56.723871    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:14:26.725101 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:14:26.725061    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:14:56.569999 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:14:56.569967    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:14:56.723774 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:14:56.723739    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:15:07.579221 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:07.579128    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:15:12.172652 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:12.172622    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:15:12.173112 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:12.172856    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kserve-container" containerID="cri-o://8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8" gracePeriod=30
Oct 30 19:15:12.173112 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:12.172911    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kube-rbac-proxy" containerID="cri-o://702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2" gracePeriod=30
Oct 30 19:15:12.926390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:12.926347    4942 generic.go:356] "Generic (PLEG): container finished" podID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerID="702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2" exitCode=0
Oct 30 19:15:12.926567 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:12.926404    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerDied","Data":"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"}
Oct 30 19:15:13.611327 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.611298    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:15:13.757364 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757271    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\") pod \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") "
Oct 30 19:15:13.757364 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757328    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\") pod \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") "
Oct 30 19:15:13.757364 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757362    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\" (UniqueName: \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\") pod \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") "
Oct 30 19:15:13.757588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757537    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\") pod \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") "
Oct 30 19:15:13.757647 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757629    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\") pod \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\" (UID: \"3a6db8ee-819b-4719-ab25-ac0f2fa59b79\") "
Oct 30 19:15:13.757777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757729    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config" pod "3a6db8ee-819b-4719-ab25-ac0f2fa59b79" (UID: "3a6db8ee-819b-4719-ab25-ac0f2fa59b79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.757899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757884    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.757963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.757897    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert" pod "3a6db8ee-819b-4719-ab25-ac0f2fa59b79" (UID: "3a6db8ee-819b-4719-ab25-ac0f2fa59b79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.759564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.759538    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls" pod "3a6db8ee-819b-4719-ab25-ac0f2fa59b79" (UID: "3a6db8ee-819b-4719-ab25-ac0f2fa59b79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:13.759837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.759823    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd" pod "3a6db8ee-819b-4719-ab25-ac0f2fa59b79" (UID: "3a6db8ee-819b-4719-ab25-ac0f2fa59b79"). InnerVolumeSpecName "kube-api-access-x94qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.762953 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.762923    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location" pod "3a6db8ee-819b-4719-ab25-ac0f2fa59b79" (UID: "3a6db8ee-819b-4719-ab25-ac0f2fa59b79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.859162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.859107    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x94qd\" (UniqueName: \"kubernetes.io/projected/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kube-api-access-x94qd\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.859162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.859156    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.859162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.859168    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.859162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.859176    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6db8ee-819b-4719-ab25-ac0f2fa59b79-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.932672 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.932634    4942 generic.go:356] "Generic (PLEG): container finished" podID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerID="8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8" exitCode=0
Oct 30 19:15:13.932672 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.932674    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerDied","Data":"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"}
Oct 30 19:15:13.932890 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.932723    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"
Oct 30 19:15:13.932890 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.932745    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p" event={"ID":"3a6db8ee-819b-4719-ab25-ac0f2fa59b79","Type":"ContainerDied","Data":"868e1347048c7d43f391293c380b35ff68fa337ba2244a28becc78e793a30191"}
Oct 30 19:15:13.932890 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.932772    4942 scope.go:122] "RemoveContainer" containerID="702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"
Oct 30 19:15:13.941591 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.941570    4942 scope.go:122] "RemoveContainer" containerID="8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"
Oct 30 19:15:13.949186 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.949160    4942 scope.go:122] "RemoveContainer" containerID="70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"
Oct 30 19:15:13.956542 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.956524    4942 scope.go:122] "RemoveContainer" containerID="702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"
Oct 30 19:15:13.956781 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:15:13.956761    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2\": container with ID starting with 702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2 not found: ID does not exist" containerID="702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"
Oct 30 19:15:13.956848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.956789    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2"} err="failed to get container status \"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2\": rpc error: code = NotFound desc = could not find container \"702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2\": container with ID starting with 702efa279c3183834fcb7f11f509135cc12bc06b25db842b298c9d21ce87dfe2 not found: ID does not exist"
Oct 30 19:15:13.956848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.956806    4942 scope.go:122] "RemoveContainer" containerID="8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"
Oct 30 19:15:13.957071 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:15:13.957044    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8\": container with ID starting with 8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8 not found: ID does not exist" containerID="8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"
Oct 30 19:15:13.957139 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.957077    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8"} err="failed to get container status \"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8\": rpc error: code = NotFound desc = could not find container \"8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8\": container with ID starting with 8f5637151b11df55c7c382e4506ba7750466c34efa31cec54e28dcdcd90520f8 not found: ID does not exist"
Oct 30 19:15:13.957139 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.957091    4942 scope.go:122] "RemoveContainer" containerID="70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"
Oct 30 19:15:13.957319 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:15:13.957299    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761\": container with ID starting with 70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761 not found: ID does not exist" containerID="70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"
Oct 30 19:15:13.957379 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.957329    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761"} err="failed to get container status \"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761\": rpc error: code = NotFound desc = could not find container \"70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761\": container with ID starting with 70f2a28988556ffd1489fe6f4f2e483330dea6ede3eee5c75f2244bd68796761 not found: ID does not exist"
Oct 30 19:15:13.957419 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.957377    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:15:13.959945 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:13.959915    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-variants-predictor-d5c7f5ccf-5lm4p"]
Oct 30 19:15:15.541762 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.541727    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" path="/var/lib/kubelet/pods/3a6db8ee-819b-4719-ab25-ac0f2fa59b79/volumes"
Oct 30 19:15:15.706122 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706091    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:15:15.706418 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706405    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="storage-initializer"
Oct 30 19:15:15.706459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706418    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="storage-initializer"
Oct 30 19:15:15.706459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706431    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kube-rbac-proxy"
Oct 30 19:15:15.706459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706436    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kube-rbac-proxy"
Oct 30 19:15:15.706459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706458    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kserve-container"
Oct 30 19:15:15.706585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706464    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kserve-container"
Oct 30 19:15:15.706585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706512    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kube-rbac-proxy"
Oct 30 19:15:15.706585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.706523    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6db8ee-819b-4719-ab25-ac0f2fa59b79" containerName="kserve-container"
Oct 30 19:15:15.711280 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.711263    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.723844 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.723823    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:15:15.723983 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.723947    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 30 19:15:15.723983 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.723959    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 30 19:15:15.723983 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.723965    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:15:15.724158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.724037    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:15:15.724158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.724058    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:15:15.724158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.724134    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:15:15.727608 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.727587    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:15:15.875390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.875301    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.875390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.875340    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sgc\" (UniqueName: \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.875390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.875363    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.875616 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.875502    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.875616 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.875560    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.976966    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977034    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sgc\" (UniqueName: \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977067    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977105    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977179    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977546 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977524    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977761    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.977828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.977786    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.979694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.979672    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:15.990411 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:15.990384    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sgc\" (UniqueName: \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\") pod \"isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:16.028747 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:16.028709    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:16.309494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:16.309469    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:15:16.312424 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:15:16.312386    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0dc1c6_7cbc_4faa_8234_a8dbf7f2f99c.slice/crio-19f62b732b9d927ff326eafdcb691c5a58d4fe56c30e37088b0fc4db78069f4b WatchSource:0}: Error finding container 19f62b732b9d927ff326eafdcb691c5a58d4fe56c30e37088b0fc4db78069f4b: Status 404 returned error can't find the container with id 19f62b732b9d927ff326eafdcb691c5a58d4fe56c30e37088b0fc4db78069f4b
Oct 30 19:15:16.947450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:16.947413    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerStarted","Data":"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"}
Oct 30 19:15:16.947862 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:16.947473    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerStarted","Data":"19f62b732b9d927ff326eafdcb691c5a58d4fe56c30e37088b0fc4db78069f4b"}
Oct 30 19:15:22.971816 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:22.971779    4942 generic.go:356] "Generic (PLEG): container finished" podID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerID="c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2" exitCode=0
Oct 30 19:15:22.972233 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:22.971851    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerDied","Data":"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"}
Oct 30 19:15:23.979974 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:23.979934    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerStarted","Data":"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"}
Oct 30 19:15:23.980419 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:23.980034    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerStarted","Data":"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"}
Oct 30 19:15:23.980419 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:23.980288    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:24.014570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:24.014514    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" podStartSLOduration=2.886050997 podStartE2EDuration="9.014495071s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.128444074s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:24.014330242 +0000 UTC m=+2310.592425935" watchObservedRunningTime="2026-10-30 19:15:24.014495071 +0000 UTC m=+2310.592590764"
Oct 30 19:15:24.985445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:24.985411    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:26.726643 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:15:26.726589    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:15:30.995517 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:15:30.995481    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:15:56.727902 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:15:56.727859    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:16:26.728644 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:16:26.728588    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:16:53.752128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:16:53.751934    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:16:53.757138 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:16:53.755130    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:16:56.570315 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:16:56.570286    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:16:56.725416 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:16:56.725374    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:17:07.580220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:07.580170    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:17:12.269278 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:12.269240    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:17:12.269762 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:12.269552    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kserve-container" containerID="cri-o://5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150" gracePeriod=30
Oct 30 19:17:12.270362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:12.269940    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kube-rbac-proxy" containerID="cri-o://8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41" gracePeriod=30
Oct 30 19:17:12.458769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:12.458728    4942 generic.go:356] "Generic (PLEG): container finished" podID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerID="8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41" exitCode=0
Oct 30 19:17:12.458940 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:12.458806    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerDied","Data":"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"}
Oct 30 19:17:13.822751 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.822726    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:17:13.939747 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.939719    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\" (UniqueName: \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\") pod \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") "
Oct 30 19:17:13.939928 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.939769    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\") pod \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") "
Oct 30 19:17:13.939928 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.939815    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\") pod \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") "
Oct 30 19:17:13.940086 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.939940    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\") pod \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") "
Oct 30 19:17:13.940086 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.940039    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\") pod \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\" (UID: \"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c\") "
Oct 30 19:17:13.940190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.940140    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert" pod "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" (UID: "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:13.940405 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.940384    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.940503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.940428    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config" pod "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" (UID: "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:13.942085 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.942059    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc" pod "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" (UID: "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c"). InnerVolumeSpecName "kube-api-access-v6sgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:13.942179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.942098    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls" pod "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" (UID: "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:13.944579 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:13.944558    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location" pod "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" (UID: "dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.041443 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.041401    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.041443 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.041435    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sgc\" (UniqueName: \"kubernetes.io/projected/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kube-api-access-v6sgc\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.041443 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.041446    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.041680 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.041456    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.469930 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.469891    4942 generic.go:356] "Generic (PLEG): container finished" podID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerID="5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150" exitCode=0
Oct 30 19:17:14.470120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.469960    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"
Oct 30 19:17:14.470120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.469981    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerDied","Data":"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"}
Oct 30 19:17:14.470120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.470058    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc" event={"ID":"dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c","Type":"ContainerDied","Data":"19f62b732b9d927ff326eafdcb691c5a58d4fe56c30e37088b0fc4db78069f4b"}
Oct 30 19:17:14.470120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.470086    4942 scope.go:122] "RemoveContainer" containerID="8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"
Oct 30 19:17:14.478933 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.478911    4942 scope.go:122] "RemoveContainer" containerID="5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"
Oct 30 19:17:14.486168 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.486151    4942 scope.go:122] "RemoveContainer" containerID="c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"
Oct 30 19:17:14.493174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.493136    4942 scope.go:122] "RemoveContainer" containerID="8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"
Oct 30 19:17:14.493485 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:17:14.493458    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41\": container with ID starting with 8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41 not found: ID does not exist" containerID="8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"
Oct 30 19:17:14.493578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.493510    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41"} err="failed to get container status \"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41\": rpc error: code = NotFound desc = could not find container \"8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41\": container with ID starting with 8bd5eee26933ccd5253361a125bbbb0407794ee55f405fea9481841ec870ae41 not found: ID does not exist"
Oct 30 19:17:14.493578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.493534    4942 scope.go:122] "RemoveContainer" containerID="5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"
Oct 30 19:17:14.494049 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:17:14.493996    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150\": container with ID starting with 5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150 not found: ID does not exist" containerID="5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"
Oct 30 19:17:14.494152 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.494054    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150"} err="failed to get container status \"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150\": rpc error: code = NotFound desc = could not find container \"5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150\": container with ID starting with 5c8f79d4b323aeac3221dc1eb71d3b49783e0ce79175f56a5b84d7357717a150 not found: ID does not exist"
Oct 30 19:17:14.494152 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.494075    4942 scope.go:122] "RemoveContainer" containerID="c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"
Oct 30 19:17:14.494430 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:17:14.494407    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2\": container with ID starting with c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2 not found: ID does not exist" containerID="c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"
Oct 30 19:17:14.494479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.494440    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2"} err="failed to get container status \"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2\": rpc error: code = NotFound desc = could not find container \"c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2\": container with ID starting with c2ce324eadeda6cb3501cca32bbe8ecefdb0aec3e47c795524af8d6b11b01dd2 not found: ID does not exist"
Oct 30 19:17:14.495503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.495484    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:17:14.503335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:14.503313    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-v2-noslash-predictor-6b4c85d4b4-6vdsc"]
Oct 30 19:17:15.541788 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.541752    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" path="/var/lib/kubelet/pods/dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c/volumes"
Oct 30 19:17:15.920515 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920428    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:17:15.920863 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920844    4942 cpu_manager.go:414] "containerMap: removing container" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="storage-initializer"
Oct 30 19:17:15.920950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920865    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="storage-initializer"
Oct 30 19:17:15.920950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920878    4942 cpu_manager.go:414] "containerMap: removing container" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kserve-container"
Oct 30 19:17:15.920950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920885    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kserve-container"
Oct 30 19:17:15.920950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920901    4942 cpu_manager.go:414] "containerMap: removing container" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kube-rbac-proxy"
Oct 30 19:17:15.920950 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.920909    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kube-rbac-proxy"
Oct 30 19:17:15.921236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.921000    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kserve-container"
Oct 30 19:17:15.921236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.921029    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0dc1c6-7cbc-4faa-8234-a8dbf7f2f99c" containerName="kube-rbac-proxy"
Oct 30 19:17:15.926989 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.926708    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:15.938314 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.938292    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:17:15.946660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946620    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:17:15.946769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946669    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:17:15.946769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946685    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 30 19:17:15.946769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946697    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:17:15.946769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946709    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:17:15.946769 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.946719    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 30 19:17:15.947996 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:15.947978    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:17:16.058176 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.058133    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.058176 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.058181    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.058401 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.058263    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.058401 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.058325    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.058478 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.058403    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdgb\" (UniqueName: \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.159441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159383    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.159648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159476    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.159648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159513    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdgb\" (UniqueName: \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.159648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159555    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.159648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159587    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.160027 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.159980    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.160290 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.160269    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.160360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.160286    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.162171 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.162145    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.172931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.172874    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdgb\" (UniqueName: \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\") pod \"isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.244699 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.244669    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:16.527368 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:16.527339    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:17:16.530530 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:17:16.530489    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c6a63e_b37a_41c0_b2f3_61d282e67929.slice/crio-8f63c3cd9a73cbcdd610cd30d849a9f28d15c53fb8421ee36293a82d66505f49 WatchSource:0}: Error finding container 8f63c3cd9a73cbcdd610cd30d849a9f28d15c53fb8421ee36293a82d66505f49: Status 404 returned error can't find the container with id 8f63c3cd9a73cbcdd610cd30d849a9f28d15c53fb8421ee36293a82d66505f49
Oct 30 19:17:17.486689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:17.486655    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerStarted","Data":"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"}
Oct 30 19:17:17.487101 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:17.486712    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerStarted","Data":"8f63c3cd9a73cbcdd610cd30d849a9f28d15c53fb8421ee36293a82d66505f49"}
Oct 30 19:17:26.526480 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:26.526445    4942 generic.go:356] "Generic (PLEG): container finished" podID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerID="99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90" exitCode=0
Oct 30 19:17:26.526926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:26.526526    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerDied","Data":"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"}
Oct 30 19:17:26.727196 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:17:26.727137    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:17:27.533494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.533456    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerStarted","Data":"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"}
Oct 30 19:17:27.533928 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.533551    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerStarted","Data":"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"}
Oct 30 19:17:27.533928 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.533832    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:27.534062 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.534038    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:27.535524 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.535498    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 19:17:27.562499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:27.562441    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podStartSLOduration=3.076158056 podStartE2EDuration="12.562426733s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.486268677s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:27.560174163 +0000 UTC m=+2434.138269855" watchObservedRunningTime="2026-10-30 19:17:27.562426733 +0000 UTC m=+2434.140522424"
Oct 30 19:17:28.538806 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:28.538765    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 19:17:33.544394 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:33.544356    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:33.545045 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:33.544999    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 19:17:43.545790 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:43.545747    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:17:56.724383 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:17:56.724346    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:17:59.623281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:59.623242    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:17:59.623845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:59.623476    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container" containerID="cri-o://37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11" gracePeriod=30
Oct 30 19:17:59.623845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:17:59.623521    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kube-rbac-proxy" containerID="cri-o://da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2" gracePeriod=30
Oct 30 19:18:00.695762 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:00.695721    4942 generic.go:356] "Generic (PLEG): container finished" podID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerID="da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2" exitCode=0
Oct 30 19:18:00.696187 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:00.695791    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerDied","Data":"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"}
Oct 30 19:18:01.587482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.587451    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:18:01.648220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648183    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\") pod \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") "
Oct 30 19:18:01.648392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648247    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\") pod \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") "
Oct 30 19:18:01.648392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648279    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\") pod \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") "
Oct 30 19:18:01.648392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648341    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\" (UniqueName: \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\") pod \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") "
Oct 30 19:18:01.648392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648371    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\") pod \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\" (UID: \"d3c6a63e-b37a-41c0-b2f3-61d282e67929\") "
Oct 30 19:18:01.648754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648715    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert" pod "d3c6a63e-b37a-41c0-b2f3-61d282e67929" (UID: "d3c6a63e-b37a-41c0-b2f3-61d282e67929"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:01.648875 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.648804    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config" pod "d3c6a63e-b37a-41c0-b2f3-61d282e67929" (UID: "d3c6a63e-b37a-41c0-b2f3-61d282e67929"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:01.649373 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.649345    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location" pod "d3c6a63e-b37a-41c0-b2f3-61d282e67929" (UID: "d3c6a63e-b37a-41c0-b2f3-61d282e67929"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:01.650571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.650544    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb" pod "d3c6a63e-b37a-41c0-b2f3-61d282e67929" (UID: "d3c6a63e-b37a-41c0-b2f3-61d282e67929"). InnerVolumeSpecName "kube-api-access-xrdgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:01.650658 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.650573    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls" pod "d3c6a63e-b37a-41c0-b2f3-61d282e67929" (UID: "d3c6a63e-b37a-41c0-b2f3-61d282e67929"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:01.702429 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.702391    4942 generic.go:356] "Generic (PLEG): container finished" podID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerID="37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11" exitCode=0
Oct 30 19:18:01.702901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.702472    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerDied","Data":"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"}
Oct 30 19:18:01.702901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.702503    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"
Oct 30 19:18:01.702901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.702541    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd" event={"ID":"d3c6a63e-b37a-41c0-b2f3-61d282e67929","Type":"ContainerDied","Data":"8f63c3cd9a73cbcdd610cd30d849a9f28d15c53fb8421ee36293a82d66505f49"}
Oct 30 19:18:01.702901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.702586    4942 scope.go:122] "RemoveContainer" containerID="da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"
Oct 30 19:18:01.712691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.712669    4942 scope.go:122] "RemoveContainer" containerID="37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"
Oct 30 19:18:01.720903 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.720879    4942 scope.go:122] "RemoveContainer" containerID="99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"
Oct 30 19:18:01.728982 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.728940    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:18:01.732808 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.732779    4942 scope.go:122] "RemoveContainer" containerID="da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"
Oct 30 19:18:01.733243 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:18:01.733211    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2\": container with ID starting with da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2 not found: ID does not exist" containerID="da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"
Oct 30 19:18:01.733243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733232    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-predictor-c8fb8979f-jbrxd"]
Oct 30 19:18:01.733367 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733255    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2"} err="failed to get container status \"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2\": rpc error: code = NotFound desc = could not find container \"da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2\": container with ID starting with da5e8d973c51fe7f6ea825102766e6993929fa05a9c4fbf9e4e6df210e7029a2 not found: ID does not exist"
Oct 30 19:18:01.733367 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733279    4942 scope.go:122] "RemoveContainer" containerID="37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"
Oct 30 19:18:01.733605 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:18:01.733584    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11\": container with ID starting with 37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11 not found: ID does not exist" containerID="37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"
Oct 30 19:18:01.733649 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733612    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11"} err="failed to get container status \"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11\": rpc error: code = NotFound desc = could not find container \"37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11\": container with ID starting with 37f863f86a354d6ca2089f1e3a8e24def7c611f7eb1f9fea31e59536f016ba11 not found: ID does not exist"
Oct 30 19:18:01.733649 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733628    4942 scope.go:122] "RemoveContainer" containerID="99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"
Oct 30 19:18:01.733877 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:18:01.733859    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90\": container with ID starting with 99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90 not found: ID does not exist" containerID="99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"
Oct 30 19:18:01.733931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.733882    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90"} err="failed to get container status \"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90\": rpc error: code = NotFound desc = could not find container \"99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90\": container with ID starting with 99a7a161cfda8480408e7f0f95365cfa2079aa8ad3323cc0382d08d3f846ec90 not found: ID does not exist"
Oct 30 19:18:01.749331 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.749307    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdgb\" (UniqueName: \"kubernetes.io/projected/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-api-access-xrdgb\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.749416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.749333    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.749416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.749345    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.749416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.749355    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c6a63e-b37a-41c0-b2f3-61d282e67929-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.749416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:01.749364    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c6a63e-b37a-41c0-b2f3-61d282e67929-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.519632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519599    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:18:02.519948 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519924    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container"
Oct 30 19:18:02.519948 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519939    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519960    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="storage-initializer"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519965    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="storage-initializer"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519976    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kube-rbac-proxy"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.519983    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kube-rbac-proxy"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.520073    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kserve-container"
Oct 30 19:18:02.520128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.520088    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" containerName="kube-rbac-proxy"
Oct 30 19:18:02.524872 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.524849    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.530674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.530647    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:18:02.531752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.531598    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:18:02.531752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.531654    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:02.531752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.531650    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 19:18:02.531752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.531674    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:18:02.531944 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.531928    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:18:02.532111 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.532090    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:02.538433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.538395    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:18:02.554517 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.554479    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.554681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.554538    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758qf\" (UniqueName: \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.554681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.554595    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.554681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.554615    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.554809 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.554707    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.655517 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.655479    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.655692 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.655549    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758qf\" (UniqueName: \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.655692 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.655580    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.655692 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.655608    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.655692 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.655640    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.656109 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.656082    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.656340 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.656319    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.656392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.656346    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.658135 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.658116    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.668973 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.668946    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758qf\" (UniqueName: \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\") pod \"isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:02.843205 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:02.843122    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:03.143551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.143515    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:18:03.144760 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:18:03.144734    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acef09f_2a37_49ef_a812_f151d0df8b38.slice/crio-26322254b713b516c2ba65c2482b9b20103faaaa7887f94adc9b9035e986a76b WatchSource:0}: Error finding container 26322254b713b516c2ba65c2482b9b20103faaaa7887f94adc9b9035e986a76b: Status 404 returned error can't find the container with id 26322254b713b516c2ba65c2482b9b20103faaaa7887f94adc9b9035e986a76b
Oct 30 19:18:03.232336 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.232291    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:03.241251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.241211    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.252036 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.251977    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 30 19:18:03.252199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.252171    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:03.252199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.252175    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:18:03.261681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.261641    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:03.350323 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.350284    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:18:03.350503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.350455    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:18:03.350503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.350456    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 30 19:18:03.350590 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.350541    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:03.362105 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.362060    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.362253 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.362121    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.362253 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.362221    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.362331 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.362272    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.362376 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.362357    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h45t\" (UniqueName: \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.463683 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.463652    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.463830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.463692    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.463830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.463725    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.463830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.463756    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.463830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.463788    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h45t\" (UniqueName: \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.464372 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.464348    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.464671 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.464651    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.464802 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.464773    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.466395 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.466370    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.476942 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.476909    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h45t\" (UniqueName: \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") " pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.544418 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.544382    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c6a63e-b37a-41c0-b2f3-61d282e67929" path="/var/lib/kubelet/pods/d3c6a63e-b37a-41c0-b2f3-61d282e67929/volumes"
Oct 30 19:18:03.559663 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.559619    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:03.714581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.714486    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerStarted","Data":"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"}
Oct 30 19:18:03.714581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.714554    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerStarted","Data":"26322254b713b516c2ba65c2482b9b20103faaaa7887f94adc9b9035e986a76b"}
Oct 30 19:18:03.848697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:03.848515    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:03.851346 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:18:03.851315    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d71081a_b018_474b_89c2_a13c03d57bbb.slice/crio-e565de74cfc9daa8b320b7b63695bae18418236aec150ceeac36ad1521398709 WatchSource:0}: Error finding container e565de74cfc9daa8b320b7b63695bae18418236aec150ceeac36ad1521398709: Status 404 returned error can't find the container with id e565de74cfc9daa8b320b7b63695bae18418236aec150ceeac36ad1521398709
Oct 30 19:18:04.721094 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:04.721048    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerStarted","Data":"8c8e42cef393f31dd3732196e965de8ff50aa825a87afae33dc00981d0f8a19f"}
Oct 30 19:18:04.721304 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:04.721119    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerStarted","Data":"e565de74cfc9daa8b320b7b63695bae18418236aec150ceeac36ad1521398709"}
Oct 30 19:18:07.735435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:07.735400    4942 generic.go:356] "Generic (PLEG): container finished" podID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerID="f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081" exitCode=0
Oct 30 19:18:07.735854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:07.735453    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerDied","Data":"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"}
Oct 30 19:18:08.742213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:08.742175    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerStarted","Data":"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"}
Oct 30 19:18:08.742614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:08.742251    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerStarted","Data":"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"}
Oct 30 19:18:08.742614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:08.742470    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:08.742718 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:08.742622    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:08.772179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:08.772124    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" podStartSLOduration=2.755432605 podStartE2EDuration="6.77211195s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.016679345s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:08.769686062 +0000 UTC m=+2475.347781759" watchObservedRunningTime="2026-10-30 19:18:08.77211195 +0000 UTC m=+2475.350207641"
Oct 30 19:18:09.748040 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:09.747977    4942 generic.go:356] "Generic (PLEG): container finished" podID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerID="8c8e42cef393f31dd3732196e965de8ff50aa825a87afae33dc00981d0f8a19f" exitCode=0
Oct 30 19:18:09.748553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:09.748056    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerDied","Data":"8c8e42cef393f31dd3732196e965de8ff50aa825a87afae33dc00981d0f8a19f"}
Oct 30 19:18:10.755292 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.755255    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerStarted","Data":"aa5a07e8bb9fee54c0a00ca31922413b30cec3e31dea7585d3fdc5a9ebe65b62"}
Oct 30 19:18:10.755710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.755321    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerStarted","Data":"6f531d576c71ae67c0b3750be79719576b71737b5fcd198f6b21091e60f0660c"}
Oct 30 19:18:10.755710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.755605    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:10.755821 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.755776    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:10.757135 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.757110    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 19:18:10.781740 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:10.781691    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podStartSLOduration=2.820035726 podStartE2EDuration="7.781674058s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.961638332s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:10.780811764 +0000 UTC m=+2477.358907457" watchObservedRunningTime="2026-10-30 19:18:10.781674058 +0000 UTC m=+2477.359769798"
Oct 30 19:18:11.766167 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:11.761413    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 19:18:14.754210 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:14.754175    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:16.765464 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:16.765431    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:16.765998 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:16.765972    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 19:18:26.724369 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:18:26.724329    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:18:26.767150 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:26.767122    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:38.606781 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:38.606741    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:38.607272 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:38.607041    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container" containerID="cri-o://6f531d576c71ae67c0b3750be79719576b71737b5fcd198f6b21091e60f0660c" gracePeriod=30
Oct 30 19:18:38.607272 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:38.607087    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kube-rbac-proxy" containerID="cri-o://aa5a07e8bb9fee54c0a00ca31922413b30cec3e31dea7585d3fdc5a9ebe65b62" gracePeriod=30
Oct 30 19:18:38.894086 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:38.893989    4942 generic.go:356] "Generic (PLEG): container finished" podID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerID="aa5a07e8bb9fee54c0a00ca31922413b30cec3e31dea7585d3fdc5a9ebe65b62" exitCode=0
Oct 30 19:18:38.894086 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:38.894066    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerDied","Data":"aa5a07e8bb9fee54c0a00ca31922413b30cec3e31dea7585d3fdc5a9ebe65b62"}
Oct 30 19:18:39.902512 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:39.902479    4942 generic.go:356] "Generic (PLEG): container finished" podID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerID="6f531d576c71ae67c0b3750be79719576b71737b5fcd198f6b21091e60f0660c" exitCode=0
Oct 30 19:18:39.902843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:39.902545    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerDied","Data":"6f531d576c71ae67c0b3750be79719576b71737b5fcd198f6b21091e60f0660c"}
Oct 30 19:18:39.955387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:39.955364    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:40.074488 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074399    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\") pod \"0d71081a-b018-474b-89c2-a13c03d57bbb\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") "
Oct 30 19:18:40.074488 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074442    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\") pod \"0d71081a-b018-474b-89c2-a13c03d57bbb\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") "
Oct 30 19:18:40.074488 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074470    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\") pod \"0d71081a-b018-474b-89c2-a13c03d57bbb\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") "
Oct 30 19:18:40.074488 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074490    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\") pod \"0d71081a-b018-474b-89c2-a13c03d57bbb\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") "
Oct 30 19:18:40.074828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074547    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\" (UniqueName: \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\") pod \"0d71081a-b018-474b-89c2-a13c03d57bbb\" (UID: \"0d71081a-b018-474b-89c2-a13c03d57bbb\") "
Oct 30 19:18:40.074889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074821    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config" pod "0d71081a-b018-474b-89c2-a13c03d57bbb" (UID: "0d71081a-b018-474b-89c2-a13c03d57bbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:40.074889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.074850    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert" pod "0d71081a-b018-474b-89c2-a13c03d57bbb" (UID: "0d71081a-b018-474b-89c2-a13c03d57bbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:40.075530 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.075505    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location" pod "0d71081a-b018-474b-89c2-a13c03d57bbb" (UID: "0d71081a-b018-474b-89c2-a13c03d57bbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:40.076677 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.076651    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t" pod "0d71081a-b018-474b-89c2-a13c03d57bbb" (UID: "0d71081a-b018-474b-89c2-a13c03d57bbb"). InnerVolumeSpecName "kube-api-access-7h45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:40.076731 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.076708    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls" pod "0d71081a-b018-474b-89c2-a13c03d57bbb" (UID: "0d71081a-b018-474b-89c2-a13c03d57bbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:40.175165 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.175133    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h45t\" (UniqueName: \"kubernetes.io/projected/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-api-access-7h45t\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.175165 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.175163    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.175358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.175178    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d71081a-b018-474b-89c2-a13c03d57bbb-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.175358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.175191    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d71081a-b018-474b-89c2-a13c03d57bbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.175358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.175204    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d71081a-b018-474b-89c2-a13c03d57bbb-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.908437 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.908403    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v" event={"ID":"0d71081a-b018-474b-89c2-a13c03d57bbb","Type":"ContainerDied","Data":"e565de74cfc9daa8b320b7b63695bae18418236aec150ceeac36ad1521398709"}
Oct 30 19:18:40.908852 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.908468    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"
Oct 30 19:18:40.908852 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.908477    4942 scope.go:122] "RemoveContainer" containerID="aa5a07e8bb9fee54c0a00ca31922413b30cec3e31dea7585d3fdc5a9ebe65b62"
Oct 30 19:18:40.918214 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.918197    4942 scope.go:122] "RemoveContainer" containerID="6f531d576c71ae67c0b3750be79719576b71737b5fcd198f6b21091e60f0660c"
Oct 30 19:18:40.925625 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.925608    4942 scope.go:122] "RemoveContainer" containerID="8c8e42cef393f31dd3732196e965de8ff50aa825a87afae33dc00981d0f8a19f"
Oct 30 19:18:40.933843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.933823    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:40.937922 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:40.937897    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-autogluon-ts-v1-noslash-predictor-8dd865dd4-l974v"]
Oct 30 19:18:41.542445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:41.542411    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" path="/var/lib/kubelet/pods/0d71081a-b018-474b-89c2-a13c03d57bbb/volumes"
Oct 30 19:18:42.115891 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.115855    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:18:42.116303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116255    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container"
Oct 30 19:18:42.116303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116272    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container"
Oct 30 19:18:42.116303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116282    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="storage-initializer"
Oct 30 19:18:42.116303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116287    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="storage-initializer"
Oct 30 19:18:42.116303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116306    4942 cpu_manager.go:414] "containerMap: removing container" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kube-rbac-proxy"
Oct 30 19:18:42.116513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116311    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kube-rbac-proxy"
Oct 30 19:18:42.116513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116375    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kserve-container"
Oct 30 19:18:42.116513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.116385    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d71081a-b018-474b-89c2-a13c03d57bbb" containerName="kube-rbac-proxy"
Oct 30 19:18:42.121065 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.121048    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.126059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.126038    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:18:42.126175 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.126082    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 19:18:42.127490 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.127405    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:18:42.129343 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.129325    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:18:42.131258 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.131240    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:18:42.197328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.197298    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:18:42.200895 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.200878    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.221519 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.221496    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:18:42.225880 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.225852    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:42.225975 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.225892    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:42.226107 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.226090    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 19:18:42.226161 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.226149    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 30 19:18:42.294156 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294129    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.294328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294171    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptqb\" (UniqueName: \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.294328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294303    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.294447 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294344    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.294447 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294370    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.294447 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294417    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59f5j\" (UniqueName: \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.294564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294508    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.294564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294542    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.294675 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294561    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.294675 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.294592    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.395648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395585    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.395648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395623    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptqb\" (UniqueName: \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395652    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395678    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395702    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395723    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59f5j\" (UniqueName: \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395766    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.395829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395803    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.396169 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.395947    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.396169 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396037    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.396342 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396320    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.396569 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396541    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.396569 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396553    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.396737 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396577    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.396781 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.396747    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.397053 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.397036    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.398710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.398686    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.398710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.398703    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.409783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.409755    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59f5j\" (UniqueName: \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\") pod \"isvc-canary-create-v2-predictor-fff4fc46f-fpdrk\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.410153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.410134    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptqb\" (UniqueName: \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\") pod \"isvc-canary-create-predictor-9b6f958d8-gg7xg\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") " pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.439174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.439149    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:42.517754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.517723    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:42.718358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.718326    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:18:42.721498 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:18:42.721473    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9270b6c_7a87_4908_878a_c76282df2bb0.slice/crio-e3fdb5fa6b1968528e8a4c0748f4853447d14742c5e23623f3cade82d9baa4e8 WatchSource:0}: Error finding container e3fdb5fa6b1968528e8a4c0748f4853447d14742c5e23623f3cade82d9baa4e8: Status 404 returned error can't find the container with id e3fdb5fa6b1968528e8a4c0748f4853447d14742c5e23623f3cade82d9baa4e8
Oct 30 19:18:42.796922 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.796903    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:18:42.799319 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:18:42.799295    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb28787_e47a_411b_89e2_1107b19179d7.slice/crio-29a0a13650c74d2bc5b01313eadd86f066bcacb9a2b6013e6ab65e5283da6ad9 WatchSource:0}: Error finding container 29a0a13650c74d2bc5b01313eadd86f066bcacb9a2b6013e6ab65e5283da6ad9: Status 404 returned error can't find the container with id 29a0a13650c74d2bc5b01313eadd86f066bcacb9a2b6013e6ab65e5283da6ad9
Oct 30 19:18:42.922480 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.922436    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerStarted","Data":"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"}
Oct 30 19:18:42.922651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.922514    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerStarted","Data":"29a0a13650c74d2bc5b01313eadd86f066bcacb9a2b6013e6ab65e5283da6ad9"}
Oct 30 19:18:42.924103 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.924080    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerStarted","Data":"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"}
Oct 30 19:18:42.924230 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:42.924123    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerStarted","Data":"e3fdb5fa6b1968528e8a4c0748f4853447d14742c5e23623f3cade82d9baa4e8"}
Oct 30 19:18:44.844631 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:44.844601    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:46.943753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:46.943722    4942 generic.go:356] "Generic (PLEG): container finished" podID="ffb28787-e47a-411b-89e2-1107b19179d7" containerID="036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a" exitCode=0
Oct 30 19:18:46.944146 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:46.943799    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerDied","Data":"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"}
Oct 30 19:18:46.945396 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:46.945376    4942 generic.go:356] "Generic (PLEG): container finished" podID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerID="a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497" exitCode=0
Oct 30 19:18:46.945499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:46.945449    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerDied","Data":"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"}
Oct 30 19:18:47.953064 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.952993    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerStarted","Data":"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"}
Oct 30 19:18:47.953513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.953105    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerStarted","Data":"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"}
Oct 30 19:18:47.953513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.953425    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:47.953513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.953485    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:47.955124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.955094    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:18:47.955308 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.955289    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerStarted","Data":"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"}
Oct 30 19:18:47.955387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.955332    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerStarted","Data":"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"}
Oct 30 19:18:47.955667 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.955646    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:47.955770 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.955690    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:47.956656 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.956630    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:18:47.983220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:47.983176    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podStartSLOduration=2.158710563 podStartE2EDuration="5.983163578s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824453015s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:47.982161677 +0000 UTC m=+2514.560257370" watchObservedRunningTime="2026-10-30 19:18:47.983163578 +0000 UTC m=+2514.561259270"
Oct 30 19:18:48.010894 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:48.010848    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podStartSLOduration=1.992850937 podStartE2EDuration="6.010840007s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.01798907s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:48.009891829 +0000 UTC m=+2514.587987544" watchObservedRunningTime="2026-10-30 19:18:48.010840007 +0000 UTC m=+2514.588935698"
Oct 30 19:18:48.959765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:48.959730    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:18:48.960244 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:48.959927    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:18:52.931256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:52.931224    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:18:52.931642 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:52.931459    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kserve-container" containerID="cri-o://ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd" gracePeriod=30
Oct 30 19:18:52.931642 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:52.931540    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy" containerID="cri-o://665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa" gracePeriod=30
Oct 30 19:18:53.964116 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.964090    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:18:53.964569 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.964547    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:18:53.964715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.964585    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:18:53.964989 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.964966    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:18:53.988725 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.988703    4942 generic.go:356] "Generic (PLEG): container finished" podID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerID="665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa" exitCode=0
Oct 30 19:18:53.988840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:53.988778    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerDied","Data":"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"}
Oct 30 19:18:54.749663 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:54.749614    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Oct 30 19:18:56.724487 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:18:56.724444    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:18:59.748700 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.748653    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Oct 30 19:18:59.894253 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.894226    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:18:59.941897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.941866    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\" (UniqueName: \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\") pod \"5acef09f-2a37-49ef-a812-f151d0df8b38\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") "
Oct 30 19:18:59.942049 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.941954    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\") pod \"5acef09f-2a37-49ef-a812-f151d0df8b38\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") "
Oct 30 19:18:59.942049 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942022    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\") pod \"5acef09f-2a37-49ef-a812-f151d0df8b38\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") "
Oct 30 19:18:59.942149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942078    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\") pod \"5acef09f-2a37-49ef-a812-f151d0df8b38\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") "
Oct 30 19:18:59.942149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942113    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\") pod \"5acef09f-2a37-49ef-a812-f151d0df8b38\" (UID: \"5acef09f-2a37-49ef-a812-f151d0df8b38\") "
Oct 30 19:18:59.942427 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942308    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert" pod "5acef09f-2a37-49ef-a812-f151d0df8b38" (UID: "5acef09f-2a37-49ef-a812-f151d0df8b38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:59.942564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942538    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location" pod "5acef09f-2a37-49ef-a812-f151d0df8b38" (UID: "5acef09f-2a37-49ef-a812-f151d0df8b38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:59.942634 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.942538    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config" pod "5acef09f-2a37-49ef-a812-f151d0df8b38" (UID: "5acef09f-2a37-49ef-a812-f151d0df8b38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:59.944076 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.944040    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf" pod "5acef09f-2a37-49ef-a812-f151d0df8b38" (UID: "5acef09f-2a37-49ef-a812-f151d0df8b38"). InnerVolumeSpecName "kube-api-access-758qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:59.944351 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:18:59.944329    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls" pod "5acef09f-2a37-49ef-a812-f151d0df8b38" (UID: "5acef09f-2a37-49ef-a812-f151d0df8b38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:00.018627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.018597    4942 generic.go:356] "Generic (PLEG): container finished" podID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerID="ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd" exitCode=0
Oct 30 19:19:00.018794 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.018678    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"
Oct 30 19:19:00.018794 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.018682    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerDied","Data":"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"}
Oct 30 19:19:00.018912 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.018804    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9" event={"ID":"5acef09f-2a37-49ef-a812-f151d0df8b38","Type":"ContainerDied","Data":"26322254b713b516c2ba65c2482b9b20103faaaa7887f94adc9b9035e986a76b"}
Oct 30 19:19:00.018912 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.018834    4942 scope.go:122] "RemoveContainer" containerID="665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"
Oct 30 19:19:00.028353 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.028335    4942 scope.go:122] "RemoveContainer" containerID="ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"
Oct 30 19:19:00.035842 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.035823    4942 scope.go:122] "RemoveContainer" containerID="f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"
Oct 30 19:19:00.042748 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.042726    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758qf\" (UniqueName: \"kubernetes.io/projected/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-api-access-758qf\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:00.042840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.042749    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:00.042840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.042761    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5acef09f-2a37-49ef-a812-f151d0df8b38-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:00.042840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.042771    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5acef09f-2a37-49ef-a812-f151d0df8b38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:00.042840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.042779    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5acef09f-2a37-49ef-a812-f151d0df8b38-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:00.043248 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.043223    4942 scope.go:122] "RemoveContainer" containerID="665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"
Oct 30 19:19:00.043551 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:19:00.043528    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa\": container with ID starting with 665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa not found: ID does not exist" containerID="665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"
Oct 30 19:19:00.043660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.043558    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa"} err="failed to get container status \"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa\": rpc error: code = NotFound desc = could not find container \"665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa\": container with ID starting with 665aeaff7d35af66af495a2d42d6a1720b3567fe41826170041ab5ea7b6b0faa not found: ID does not exist"
Oct 30 19:19:00.043660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.043575    4942 scope.go:122] "RemoveContainer" containerID="ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"
Oct 30 19:19:00.043841 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:19:00.043823    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd\": container with ID starting with ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd not found: ID does not exist" containerID="ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"
Oct 30 19:19:00.043894 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.043846    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd"} err="failed to get container status \"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd\": rpc error: code = NotFound desc = could not find container \"ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd\": container with ID starting with ec855a93157499ba7b2c46ef44f4e0d1c74b5eac0a7387ea73129bdc2cac10bd not found: ID does not exist"
Oct 30 19:19:00.043894 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.043860    4942 scope.go:122] "RemoveContainer" containerID="f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"
Oct 30 19:19:00.044145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.044126    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:19:00.044216 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:19:00.044149    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081\": container with ID starting with f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081 not found: ID does not exist" containerID="f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"
Oct 30 19:19:00.044216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.044166    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081"} err="failed to get container status \"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081\": rpc error: code = NotFound desc = could not find container \"f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081\": container with ID starting with f5e125d65ccfa5c3359db4b24c4659d1dcaa240e5fcdbea770b84fc4bfb18081 not found: ID does not exist"
Oct 30 19:19:00.048329 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:00.048310    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-mlserver-predictor-78bd55c845-xktg9"]
Oct 30 19:19:01.541664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:01.541634    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" path="/var/lib/kubelet/pods/5acef09f-2a37-49ef-a812-f151d0df8b38/volumes"
Oct 30 19:19:02.334072 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334040    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:19:02.334392 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334380    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy"
Oct 30 19:19:02.334439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334393    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy"
Oct 30 19:19:02.334439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334409    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kserve-container"
Oct 30 19:19:02.334439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334414    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kserve-container"
Oct 30 19:19:02.334439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334435    4942 cpu_manager.go:414] "containerMap: removing container" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="storage-initializer"
Oct 30 19:19:02.334439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334440    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="storage-initializer"
Oct 30 19:19:02.334585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334493    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kube-rbac-proxy"
Oct 30 19:19:02.334585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.334503    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acef09f-2a37-49ef-a812-f151d0df8b38" containerName="kserve-container"
Oct 30 19:19:02.338994 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.338975    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.348107 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.348076    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 19:19:02.353592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.353572    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:19:02.356702 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.356681    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:19:02.356702 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.356688    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:19:02.356848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.356738    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:19:02.356848 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.356772    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:19:02.362644 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.362625    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.362726 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.362675    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.362780 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.362760    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gxz\" (UniqueName: \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.362814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.362788    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.362856 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.362824    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.447915 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.447889    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:19:02.448141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.448124    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:19:02.463454 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463433    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.463536 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463478    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.463536 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463504    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gxz\" (UniqueName: \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.463536 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463523    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.463697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463647    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.463841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.463821    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.464167 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.464149    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.464252 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.464212    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.466048 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.466030    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.476544 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.476514    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gxz\" (UniqueName: \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\") pod \"xgboost-v2-mlserver-predictor-b885bd58d-b4wm7\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") " pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.658273 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.658189    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:02.941088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:02.941061    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:19:02.944598 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:19:02.944571    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631ed918_9a8a_473a_88c6_257894741303.slice/crio-9e83af524b86fc3e3d56e8ae5fb60e47bd9778170f8f0ff7c3039d207f5b5687 WatchSource:0}: Error finding container 9e83af524b86fc3e3d56e8ae5fb60e47bd9778170f8f0ff7c3039d207f5b5687: Status 404 returned error can't find the container with id 9e83af524b86fc3e3d56e8ae5fb60e47bd9778170f8f0ff7c3039d207f5b5687
Oct 30 19:19:03.035047 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:03.034997    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerStarted","Data":"cf38dd2d8dca81dd4dd5951a73cbd510236b0fe2c243b5e93b0ee50d3940e7a2"}
Oct 30 19:19:03.035171 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:03.035101    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerStarted","Data":"9e83af524b86fc3e3d56e8ae5fb60e47bd9778170f8f0ff7c3039d207f5b5687"}
Oct 30 19:19:03.964802 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:03.964759    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:03.965298 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:03.964890    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:07.054439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:07.054405    4942 generic.go:356] "Generic (PLEG): container finished" podID="631ed918-9a8a-473a-88c6-257894741303" containerID="cf38dd2d8dca81dd4dd5951a73cbd510236b0fe2c243b5e93b0ee50d3940e7a2" exitCode=0
Oct 30 19:19:07.054921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:07.054450    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerDied","Data":"cf38dd2d8dca81dd4dd5951a73cbd510236b0fe2c243b5e93b0ee50d3940e7a2"}
Oct 30 19:19:08.060962 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:08.060930    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerStarted","Data":"1f830a8691ee1c2127fb5cfbc2808a79cf7fa4b8735eca65b46310b749ad44f4"}
Oct 30 19:19:08.061463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:08.061000    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerStarted","Data":"7dabf3326dad4027a2bbed1bfefbad0d51041d2e524305eb9c42b2e1f3390f68"}
Oct 30 19:19:08.061463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:08.061250    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:08.061463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:08.061390    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:08.089989 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:08.089943    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podStartSLOduration=2.275307671 podStartE2EDuration="6.089931421s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81462375s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:08.087346783 +0000 UTC m=+2534.665442474" watchObservedRunningTime="2026-10-30 19:19:08.089931421 +0000 UTC m=+2534.668027110"
Oct 30 19:19:13.964723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:13.964680    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:13.965128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:13.964966    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:14.070193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:14.070164    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:23.965215 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:23.965175    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:23.965809 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:23.965186    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:26.723505 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:19:26.723460    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:19:33.964727 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:33.964639    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:33.965208 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:33.964881    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:43.965306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:43.965266    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:43.965669 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:43.965267    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:44.074280 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:44.074250    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:52.665688 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:52.665649    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:19:52.666231 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:52.665865    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kserve-container" containerID="cri-o://7dabf3326dad4027a2bbed1bfefbad0d51041d2e524305eb9c42b2e1f3390f68" gracePeriod=30
Oct 30 19:19:52.666231 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:52.665929    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy" containerID="cri-o://1f830a8691ee1c2127fb5cfbc2808a79cf7fa4b8735eca65b46310b749ad44f4" gracePeriod=30
Oct 30 19:19:53.261421 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:53.261387    4942 generic.go:356] "Generic (PLEG): container finished" podID="631ed918-9a8a-473a-88c6-257894741303" containerID="1f830a8691ee1c2127fb5cfbc2808a79cf7fa4b8735eca65b46310b749ad44f4" exitCode=0
Oct 30 19:19:53.261598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:53.261464    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerDied","Data":"1f830a8691ee1c2127fb5cfbc2808a79cf7fa4b8735eca65b46310b749ad44f4"}
Oct 30 19:19:53.964750 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:53.964711    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:19:53.965179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:53.964889    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:19:54.065657 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:54.065614    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 19:19:55.112182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:55.112138    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.64:8080/v2/models/xgboost-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:56.727846 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:19:56.727804    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:19:59.065633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.065593    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 19:19:59.289174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.289141    4942 generic.go:356] "Generic (PLEG): container finished" podID="631ed918-9a8a-473a-88c6-257894741303" containerID="7dabf3326dad4027a2bbed1bfefbad0d51041d2e524305eb9c42b2e1f3390f68" exitCode=0
Oct 30 19:19:59.289363 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.289206    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerDied","Data":"7dabf3326dad4027a2bbed1bfefbad0d51041d2e524305eb9c42b2e1f3390f68"}
Oct 30 19:19:59.506319 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.506289    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:19:59.526514 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526481    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\") pod \"631ed918-9a8a-473a-88c6-257894741303\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") "
Oct 30 19:19:59.526676 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526537    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\" (UniqueName: \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\") pod \"631ed918-9a8a-473a-88c6-257894741303\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") "
Oct 30 19:19:59.526676 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526570    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\" (UniqueName: \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\") pod \"631ed918-9a8a-473a-88c6-257894741303\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") "
Oct 30 19:19:59.526676 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526608    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\") pod \"631ed918-9a8a-473a-88c6-257894741303\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") "
Oct 30 19:19:59.526676 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526642    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\") pod \"631ed918-9a8a-473a-88c6-257894741303\" (UID: \"631ed918-9a8a-473a-88c6-257894741303\") "
Oct 30 19:19:59.526942 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526851    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location" pod "631ed918-9a8a-473a-88c6-257894741303" (UID: "631ed918-9a8a-473a-88c6-257894741303"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:59.527001 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526974    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config" pod "631ed918-9a8a-473a-88c6-257894741303" (UID: "631ed918-9a8a-473a-88c6-257894741303"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:59.527086 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.526998    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert" pod "631ed918-9a8a-473a-88c6-257894741303" (UID: "631ed918-9a8a-473a-88c6-257894741303"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:59.528701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.528679    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz" pod "631ed918-9a8a-473a-88c6-257894741303" (UID: "631ed918-9a8a-473a-88c6-257894741303"). InnerVolumeSpecName "kube-api-access-t8gxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:59.528796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.528700    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls" pod "631ed918-9a8a-473a-88c6-257894741303" (UID: "631ed918-9a8a-473a-88c6-257894741303"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:59.627422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.627389    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631ed918-9a8a-473a-88c6-257894741303-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.627422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.627418    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8gxz\" (UniqueName: \"kubernetes.io/projected/631ed918-9a8a-473a-88c6-257894741303-kube-api-access-t8gxz\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.627422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.627429    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631ed918-9a8a-473a-88c6-257894741303-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.627651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.627439    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.627651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:19:59.627448    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631ed918-9a8a-473a-88c6-257894741303-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.294917 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.294873    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7" event={"ID":"631ed918-9a8a-473a-88c6-257894741303","Type":"ContainerDied","Data":"9e83af524b86fc3e3d56e8ae5fb60e47bd9778170f8f0ff7c3039d207f5b5687"}
Oct 30 19:20:00.295445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.294937    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"
Oct 30 19:20:00.295445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.294974    4942 scope.go:122] "RemoveContainer" containerID="1f830a8691ee1c2127fb5cfbc2808a79cf7fa4b8735eca65b46310b749ad44f4"
Oct 30 19:20:00.303408 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.303376    4942 scope.go:122] "RemoveContainer" containerID="7dabf3326dad4027a2bbed1bfefbad0d51041d2e524305eb9c42b2e1f3390f68"
Oct 30 19:20:00.310907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.310885    4942 scope.go:122] "RemoveContainer" containerID="cf38dd2d8dca81dd4dd5951a73cbd510236b0fe2c243b5e93b0ee50d3940e7a2"
Oct 30 19:20:00.317782 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.317751    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:20:00.322453 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:00.322427    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/xgboost-v2-mlserver-predictor-b885bd58d-b4wm7"]
Oct 30 19:20:01.542719 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:01.542686    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631ed918-9a8a-473a-88c6-257894741303" path="/var/lib/kubelet/pods/631ed918-9a8a-473a-88c6-257894741303/volumes"
Oct 30 19:20:02.222524 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222480    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:20:02.222838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222826    4942 cpu_manager.go:414] "containerMap: removing container" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kserve-container"
Oct 30 19:20:02.222884 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222839    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kserve-container"
Oct 30 19:20:02.222884 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222863    4942 cpu_manager.go:414] "containerMap: removing container" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy"
Oct 30 19:20:02.222884 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222868    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy"
Oct 30 19:20:02.222884 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222880    4942 cpu_manager.go:414] "containerMap: removing container" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="storage-initializer"
Oct 30 19:20:02.223035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222886    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="storage-initializer"
Oct 30 19:20:02.223035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222944    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kserve-container"
Oct 30 19:20:02.223035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.222954    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="631ed918-9a8a-473a-88c6-257894741303" containerName="kube-rbac-proxy"
Oct 30 19:20:02.227664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.227645    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.237553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.237530    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:20:02.237553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.237540    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 30 19:20:02.238753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.238737    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:20:02.238812 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.238776    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:20:02.239017 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.238994    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:20:02.244390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.244368    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:20:02.245598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.245536    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.245729 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.245620    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6q6\" (UniqueName: \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.245729 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.245675    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.245850 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.245814    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.245910 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.245861    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.247655 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.247633    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:20:02.337393 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.337363    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:20:02.346828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.346788    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.346828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.346832    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.346898    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.346918    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6q6\" (UniqueName: \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.346938    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347449 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.347421    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.347698    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.347873 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.347849    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.349725 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.349702    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.364059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.364027    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6q6\" (UniqueName: \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\") pod \"isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.537296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.537196    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:20:02.544933 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.544908    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:02.835589 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:02.835499    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:20:02.839957 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:20:02.839924    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd334e0d0_a17f_47dd_8c69_94e66628fb9c.slice/crio-be3d6096e4f1e9a90b4bb66d4c82b0ae0b043d6df4a5964993a40e04db63e1b4 WatchSource:0}: Error finding container be3d6096e4f1e9a90b4bb66d4c82b0ae0b043d6df4a5964993a40e04db63e1b4: Status 404 returned error can't find the container with id be3d6096e4f1e9a90b4bb66d4c82b0ae0b043d6df4a5964993a40e04db63e1b4
Oct 30 19:20:03.311850 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:03.311809    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerStarted","Data":"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"}
Oct 30 19:20:03.312029 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:03.311874    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerStarted","Data":"be3d6096e4f1e9a90b4bb66d4c82b0ae0b043d6df4a5964993a40e04db63e1b4"}
Oct 30 19:20:03.965601 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:03.965557    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 19:20:07.330360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:07.330267    4942 generic.go:356] "Generic (PLEG): container finished" podID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerID="67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97" exitCode=0
Oct 30 19:20:07.330360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:07.330344    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerDied","Data":"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"}
Oct 30 19:20:07.331907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:07.331888    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:20:12.538768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:12.538732    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:20:12.740030 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:12.739975    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:20:12.740565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:12.740254    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container" containerID="cri-o://c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43" gracePeriod=30
Oct 30 19:20:12.740565 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:12.740416    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy" containerID="cri-o://aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8" gracePeriod=30
Oct 30 19:20:12.741304 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:12.741169    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:20:13.366611 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:13.366572    4942 generic.go:356] "Generic (PLEG): container finished" podID="ffb28787-e47a-411b-89e2-1107b19179d7" containerID="aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8" exitCode=0
Oct 30 19:20:13.366811 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:13.366644    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerDied","Data":"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"}
Oct 30 19:20:13.960347 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:13.960295    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 19:20:19.026868 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.026831    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:20:19.097422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097364    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\") pod \"ffb28787-e47a-411b-89e2-1107b19179d7\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") "
Oct 30 19:20:19.097632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097457    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\") pod \"ffb28787-e47a-411b-89e2-1107b19179d7\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") "
Oct 30 19:20:19.097632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097505    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\") pod \"ffb28787-e47a-411b-89e2-1107b19179d7\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") "
Oct 30 19:20:19.097632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097528    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\" (UniqueName: \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\") pod \"ffb28787-e47a-411b-89e2-1107b19179d7\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") "
Oct 30 19:20:19.097807 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097673    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\") pod \"ffb28787-e47a-411b-89e2-1107b19179d7\" (UID: \"ffb28787-e47a-411b-89e2-1107b19179d7\") "
Oct 30 19:20:19.097927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097817    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert" pod "ffb28787-e47a-411b-89e2-1107b19179d7" (UID: "ffb28787-e47a-411b-89e2-1107b19179d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:19.097927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.097897    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config" pod "ffb28787-e47a-411b-89e2-1107b19179d7" (UID: "ffb28787-e47a-411b-89e2-1107b19179d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:19.098049 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.098003    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:19.098116 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.098046    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffb28787-e47a-411b-89e2-1107b19179d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:19.098174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.098145    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location" pod "ffb28787-e47a-411b-89e2-1107b19179d7" (UID: "ffb28787-e47a-411b-89e2-1107b19179d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:19.100199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.100166    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb" pod "ffb28787-e47a-411b-89e2-1107b19179d7" (UID: "ffb28787-e47a-411b-89e2-1107b19179d7"). InnerVolumeSpecName "kube-api-access-7ptqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:19.100339 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.100269    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls" pod "ffb28787-e47a-411b-89e2-1107b19179d7" (UID: "ffb28787-e47a-411b-89e2-1107b19179d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:19.198519 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.198473    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptqb\" (UniqueName: \"kubernetes.io/projected/ffb28787-e47a-411b-89e2-1107b19179d7-kube-api-access-7ptqb\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:19.198519 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.198513    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffb28787-e47a-411b-89e2-1107b19179d7-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:19.198519 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.198523    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffb28787-e47a-411b-89e2-1107b19179d7-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:19.402083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.401709    4942 generic.go:356] "Generic (PLEG): container finished" podID="ffb28787-e47a-411b-89e2-1107b19179d7" containerID="c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43" exitCode=0
Oct 30 19:20:19.402083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.401759    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerDied","Data":"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"}
Oct 30 19:20:19.402083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.401770    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"
Oct 30 19:20:19.402083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.401822    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" event={"ID":"ffb28787-e47a-411b-89e2-1107b19179d7","Type":"ContainerDied","Data":"29a0a13650c74d2bc5b01313eadd86f066bcacb9a2b6013e6ab65e5283da6ad9"}
Oct 30 19:20:19.402083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.401862    4942 scope.go:122] "RemoveContainer" containerID="aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"
Oct 30 19:20:19.412927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.412900    4942 scope.go:122] "RemoveContainer" containerID="c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"
Oct 30 19:20:19.423380 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.423349    4942 scope.go:122] "RemoveContainer" containerID="036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"
Oct 30 19:20:19.429597 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.429560    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:20:19.435264 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.435236    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg"]
Oct 30 19:20:19.435394 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.435381    4942 scope.go:122] "RemoveContainer" containerID="aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"
Oct 30 19:20:19.435811 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:19.435779    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8\": container with ID starting with aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8 not found: ID does not exist" containerID="aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"
Oct 30 19:20:19.435885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.435827    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8"} err="failed to get container status \"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8\": rpc error: code = NotFound desc = could not find container \"aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8\": container with ID starting with aff3971fdb1e0a853281513a74cf8c81a7b0841e3c96683968711854b19c6fe8 not found: ID does not exist"
Oct 30 19:20:19.435885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.435853    4942 scope.go:122] "RemoveContainer" containerID="c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"
Oct 30 19:20:19.436254 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:19.436229    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43\": container with ID starting with c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43 not found: ID does not exist" containerID="c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"
Oct 30 19:20:19.436349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.436261    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43"} err="failed to get container status \"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43\": rpc error: code = NotFound desc = could not find container \"c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43\": container with ID starting with c3c7c025564c69d8ad21cf982a32f7fa760923e81760a415601b493498366b43 not found: ID does not exist"
Oct 30 19:20:19.436349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.436277    4942 scope.go:122] "RemoveContainer" containerID="036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"
Oct 30 19:20:19.436654 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:19.436629    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a\": container with ID starting with 036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a not found: ID does not exist" containerID="036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"
Oct 30 19:20:19.436717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.436659    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a"} err="failed to get container status \"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a\": rpc error: code = NotFound desc = could not find container \"036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a\": container with ID starting with 036727d696dc53097bf97fbb1a15fef4b85224294557113a0f0ae2fef1ce304a not found: ID does not exist"
Oct 30 19:20:19.543508 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.543469    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" path="/var/lib/kubelet/pods/ffb28787-e47a-411b-89e2-1107b19179d7/volumes"
Oct 30 19:20:19.960742 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:19.960672    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-predictor-9b6f958d8-gg7xg" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": context deadline exceeded"
Oct 30 19:20:22.429875 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:22.429836    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:20:22.430353 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:22.430184    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" containerID="cri-o://2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810" gracePeriod=30
Oct 30 19:20:22.430353 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:22.430222    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy" containerID="cri-o://590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489" gracePeriod=30
Oct 30 19:20:22.537985 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:22.537934    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 19:20:23.426929 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:23.426885    4942 generic.go:356] "Generic (PLEG): container finished" podID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerID="590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489" exitCode=0
Oct 30 19:20:23.427143 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:23.426960    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerDied","Data":"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"}
Oct 30 19:20:23.960747 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:23.960700    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 19:20:25.439851 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:25.439813    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerStarted","Data":"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"}
Oct 30 19:20:25.440423 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:25.439887    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerStarted","Data":"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"}
Oct 30 19:20:25.440423 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:25.440143    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:25.467182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:25.467124    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podStartSLOduration=1.461127541 podStartE2EDuration="23.467105506s" totalImagesPullingTime="17.887498269s" totalInitContainerRuntime="4.118479696s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:25.466174184 +0000 UTC m=+2612.044269875" watchObservedRunningTime="2026-10-30 19:20:25.467105506 +0000 UTC m=+2612.045201195"
Oct 30 19:20:26.445316 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:26.445265    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:26.446838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:26.446800    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:20:26.727875 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:26.727787    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:20:27.450749 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:27.450702    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:20:28.996673 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:28.996635    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:20:29.095897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.095856    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\" (UniqueName: \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\") pod \"e9270b6c-7a87-4908-878a-c76282df2bb0\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") "
Oct 30 19:20:29.095897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.095907    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\") pod \"e9270b6c-7a87-4908-878a-c76282df2bb0\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") "
Oct 30 19:20:29.096201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.095930    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\") pod \"e9270b6c-7a87-4908-878a-c76282df2bb0\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") "
Oct 30 19:20:29.096201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.095990    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\") pod \"e9270b6c-7a87-4908-878a-c76282df2bb0\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") "
Oct 30 19:20:29.096201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.096050    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\") pod \"e9270b6c-7a87-4908-878a-c76282df2bb0\" (UID: \"e9270b6c-7a87-4908-878a-c76282df2bb0\") "
Oct 30 19:20:29.096374 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.096304    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location" pod "e9270b6c-7a87-4908-878a-c76282df2bb0" (UID: "e9270b6c-7a87-4908-878a-c76282df2bb0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:29.096479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.096444    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert" pod "e9270b6c-7a87-4908-878a-c76282df2bb0" (UID: "e9270b6c-7a87-4908-878a-c76282df2bb0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:29.096559 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.096532    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config" pod "e9270b6c-7a87-4908-878a-c76282df2bb0" (UID: "e9270b6c-7a87-4908-878a-c76282df2bb0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:29.098382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.098303    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j" pod "e9270b6c-7a87-4908-878a-c76282df2bb0" (UID: "e9270b6c-7a87-4908-878a-c76282df2bb0"). InnerVolumeSpecName "kube-api-access-59f5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:29.098382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.098314    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls" pod "e9270b6c-7a87-4908-878a-c76282df2bb0" (UID: "e9270b6c-7a87-4908-878a-c76282df2bb0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:29.197213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.197170    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9270b6c-7a87-4908-878a-c76282df2bb0-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.197213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.197208    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.197213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.197219    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59f5j\" (UniqueName: \"kubernetes.io/projected/e9270b6c-7a87-4908-878a-c76282df2bb0-kube-api-access-59f5j\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.197213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.197228    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9270b6c-7a87-4908-878a-c76282df2bb0-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.197543 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.197238    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9270b6c-7a87-4908-878a-c76282df2bb0-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.462639 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.462602    4942 generic.go:356] "Generic (PLEG): container finished" podID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerID="2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810" exitCode=0
Oct 30 19:20:29.462828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.462665    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"
Oct 30 19:20:29.462828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.462693    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerDied","Data":"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"}
Oct 30 19:20:29.462828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.462759    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" event={"ID":"e9270b6c-7a87-4908-878a-c76282df2bb0","Type":"ContainerDied","Data":"e3fdb5fa6b1968528e8a4c0748f4853447d14742c5e23623f3cade82d9baa4e8"}
Oct 30 19:20:29.462828 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.462790    4942 scope.go:122] "RemoveContainer" containerID="590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"
Oct 30 19:20:29.472972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.472948    4942 scope.go:122] "RemoveContainer" containerID="2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"
Oct 30 19:20:29.482602 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.482576    4942 scope.go:122] "RemoveContainer" containerID="a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"
Oct 30 19:20:29.488379 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.488343    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:20:29.493423 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.493373    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk"]
Oct 30 19:20:29.494289 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.494266    4942 scope.go:122] "RemoveContainer" containerID="590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"
Oct 30 19:20:29.494742 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:29.494716    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489\": container with ID starting with 590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489 not found: ID does not exist" containerID="590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"
Oct 30 19:20:29.494793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.494758    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489"} err="failed to get container status \"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489\": rpc error: code = NotFound desc = could not find container \"590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489\": container with ID starting with 590292277bd6c27d6f14eb1cf0d536fdbb7e42e012b446bfecc6a9bedfc65489 not found: ID does not exist"
Oct 30 19:20:29.494793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.494786    4942 scope.go:122] "RemoveContainer" containerID="2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"
Oct 30 19:20:29.495128 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:29.495102    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810\": container with ID starting with 2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810 not found: ID does not exist" containerID="2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"
Oct 30 19:20:29.495213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.495134    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810"} err="failed to get container status \"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810\": rpc error: code = NotFound desc = could not find container \"2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810\": container with ID starting with 2b94eba04ad9e17761be9f0d3020a932dbddb729849c993411cc9eb3a8085810 not found: ID does not exist"
Oct 30 19:20:29.495213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.495151    4942 scope.go:122] "RemoveContainer" containerID="a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"
Oct 30 19:20:29.495402 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:29.495385    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497\": container with ID starting with a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497 not found: ID does not exist" containerID="a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"
Oct 30 19:20:29.495439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.495406    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497"} err="failed to get container status \"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497\": rpc error: code = NotFound desc = could not find container \"a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497\": container with ID starting with a308eae0edd86a9cdb49d1185c69331e2f661acac8ec66fd9c6638c798b45497 not found: ID does not exist"
Oct 30 19:20:29.542957 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.542911    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" path="/var/lib/kubelet/pods/e9270b6c-7a87-4908-878a-c76282df2bb0/volumes"
Oct 30 19:20:29.961120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:29.961046    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-create-v2-predictor-fff4fc46f-fpdrk" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": context deadline exceeded"
Oct 30 19:20:30.122715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.122676    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123079    4942 cpu_manager.go:414] "containerMap: removing container" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="storage-initializer"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123096    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="storage-initializer"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123108    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123113    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123120    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="storage-initializer"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123126    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="storage-initializer"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123152    4942 cpu_manager.go:414] "containerMap: removing container" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123157    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123165    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123169    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123181    4942 cpu_manager.go:414] "containerMap: removing container" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123185    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123248    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123256    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kube-rbac-proxy"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123262    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9270b6c-7a87-4908-878a-c76282df2bb0" containerName="kserve-container"
Oct 30 19:20:30.123468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.123270    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb28787-e47a-411b-89e2-1107b19179d7" containerName="kserve-container"
Oct 30 19:20:30.128345 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.128311    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.141113 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.140596    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:20:30.141113 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.140643    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 19:20:30.141441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.140695    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:20:30.143061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.142573    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:20:30.143061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.142608    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:20:30.143061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.142610    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:20:30.143061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.142656    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:20:30.143758 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.143731    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 19:20:30.205459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.205412    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.205459 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.205458    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.205686 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.205571    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.205686 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.205608    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.205686 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.205658    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrgk\" (UniqueName: \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.252815 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.252722    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:20:30.256674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.256648    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.272627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.272587    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:20:30.306898 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.306857    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.306898 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.306903    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307168 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:30.307060    4942 secret.go:189] Couldn't get secret e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Oct 30 19:20:30.307168 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307059    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307168 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:30.307138    4942 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls podName:a1934637-0051-4e8b-b081-4fcd6955e993 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.807114628 +0000 UTC m=+2617.385210304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls") pod "isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Oct 30 19:20:30.307349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307206    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307236    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrgk\" (UniqueName: \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307604    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307721 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307692    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.307788 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.307755    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.320715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.320671    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrgk\" (UniqueName: \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.325026 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.324980    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 19:20:30.407991 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.407951    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.408232 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.408035    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.408232 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.408130    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dt9\" (UniqueName: \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.408232 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.408164    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.408232 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.408186    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.509382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.509285    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.509382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.509341    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.509382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.509383    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dt9\" (UniqueName: \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.509670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.509414    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.509670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.509542    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.510072 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.510044    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.510161 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.510048    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.510161 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.510147    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.512255 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.512228    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.522689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.522651    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dt9\" (UniqueName: \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\") pod \"isvc-canary-promote-predictor-5d456c5878-pcn7f\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.576222 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.576176    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:30.812587 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.812424    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.816178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.816129    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7565fb54fc-t8tml\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") " pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:30.871091 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:30.871046    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:20:30.874809 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:20:30.874773    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef54acf_53ae_4b01_a802_9f74283c1c43.slice/crio-ecd45435a1b0bf38af56f164a617650895f30e94c301dc5e64cabe0175129d7d WatchSource:0}: Error finding container ecd45435a1b0bf38af56f164a617650895f30e94c301dc5e64cabe0175129d7d: Status 404 returned error can't find the container with id ecd45435a1b0bf38af56f164a617650895f30e94c301dc5e64cabe0175129d7d
Oct 30 19:20:31.051566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.051504    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:31.339782 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.339536    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:20:31.342780 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:20:31.342740    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1934637_0051_4e8b_b081_4fcd6955e993.slice/crio-62cbaa5e8491a91eb6d2e7d99d040c8d993edeaa76589280744ed547a8bdd0ad WatchSource:0}: Error finding container 62cbaa5e8491a91eb6d2e7d99d040c8d993edeaa76589280744ed547a8bdd0ad: Status 404 returned error can't find the container with id 62cbaa5e8491a91eb6d2e7d99d040c8d993edeaa76589280744ed547a8bdd0ad
Oct 30 19:20:31.476625 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.476582    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerStarted","Data":"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"}
Oct 30 19:20:31.476839 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.476651    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerStarted","Data":"62cbaa5e8491a91eb6d2e7d99d040c8d993edeaa76589280744ed547a8bdd0ad"}
Oct 30 19:20:31.478710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.478665    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerStarted","Data":"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"}
Oct 30 19:20:31.478926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:31.478748    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerStarted","Data":"ecd45435a1b0bf38af56f164a617650895f30e94c301dc5e64cabe0175129d7d"}
Oct 30 19:20:32.455461 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:32.455426    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:20:32.455940 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:32.455909    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:20:35.501897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:35.501859    4942 generic.go:356] "Generic (PLEG): container finished" podID="a1934637-0051-4e8b-b081-4fcd6955e993" containerID="2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb" exitCode=0
Oct 30 19:20:35.502368 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:35.501933    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerDied","Data":"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"}
Oct 30 19:20:35.503861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:35.503831    4942 generic.go:356] "Generic (PLEG): container finished" podID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerID="9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe" exitCode=0
Oct 30 19:20:35.503979 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:35.503908    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerDied","Data":"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"}
Oct 30 19:20:36.511336 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.511293    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerStarted","Data":"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"}
Oct 30 19:20:36.511764 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.511398    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerStarted","Data":"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"}
Oct 30 19:20:36.511764 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.511645    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:36.514058 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.514017    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerStarted","Data":"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"}
Oct 30 19:20:36.514234 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.514090    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerStarted","Data":"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"}
Oct 30 19:20:36.514385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.514365    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:36.514430 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.514413    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:36.516156 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.516120    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:20:36.538949 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.538889    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podStartSLOduration=2.547769014 podStartE2EDuration="6.538874209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.991105195s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.5385166 +0000 UTC m=+2623.116612293" watchObservedRunningTime="2026-10-30 19:20:36.538874209 +0000 UTC m=+2623.116969879"
Oct 30 19:20:36.563059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:36.562985    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podStartSLOduration=2.272003666 podStartE2EDuration="6.562969703s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.290966037s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.562395607 +0000 UTC m=+2623.140491298" watchObservedRunningTime="2026-10-30 19:20:36.562969703 +0000 UTC m=+2623.141065395"
Oct 30 19:20:37.519624 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:37.519572    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:20:37.519624 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:37.519612    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:37.521410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:37.521374    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:20:38.524822 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:38.524771    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:20:42.457021 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:42.456948    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:20:42.525083 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:42.525041    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:20:42.525760 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:42.525727    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:20:43.530256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:43.530224    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:20:43.530881 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:43.530854    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:20:52.456641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:52.456596    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:20:52.526123 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:52.526083    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:20:53.531322 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:20:53.531282    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:20:56.723925 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:20:56.723884    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:21:02.456031 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:02.455965    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:21:02.526307 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:02.526257    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:03.531351 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:03.531307    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:12.456247 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:12.456202    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:21:12.525895 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:12.525850    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:13.531462 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:13.531422    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:22.456838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:22.456795    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:21:22.525811 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:22.525775    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:23.531783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:23.531740    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:26.725414 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:21:26.725376    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:21:32.456765 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:32.456722    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 19:21:32.525732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:32.525691    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:33.531783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:33.531744    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:42.457162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:42.457128    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:21:42.526255 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:42.526218    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:43.531463 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:43.531425    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:51.537054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:51.536989    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:21:52.525804 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:52.525767    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 19:21:53.093551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.093507    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:21:53.094071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.093801    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container" containerID="cri-o://5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b" gracePeriod=30
Oct 30 19:21:53.094071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.093887    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kube-rbac-proxy" containerID="cri-o://14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b" gracePeriod=30
Oct 30 19:21:53.779188 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.779062    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:21:53.785311 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.785287    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:21:53.866296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.866260    4942 generic.go:356] "Generic (PLEG): container finished" podID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerID="14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b" exitCode=0
Oct 30 19:21:53.866457 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:53.866335    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerDied","Data":"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"}
Oct 30 19:21:56.723665 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:21:56.723621    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:21:57.239329 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.239302    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:21:57.263966 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.263941    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\") pod \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") "
Oct 30 19:21:57.264073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.263993    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\" (UniqueName: \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\") pod \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") "
Oct 30 19:21:57.264073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264036    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\") pod \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") "
Oct 30 19:21:57.264073 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264064    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\") pod \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") "
Oct 30 19:21:57.264240 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264097    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\") pod \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\" (UID: \"d334e0d0-a17f-47dd-8c69-94e66628fb9c\") "
Oct 30 19:21:57.264420 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264390    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert" pod "d334e0d0-a17f-47dd-8c69-94e66628fb9c" (UID: "d334e0d0-a17f-47dd-8c69-94e66628fb9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:57.264504 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264422    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config" pod "d334e0d0-a17f-47dd-8c69-94e66628fb9c" (UID: "d334e0d0-a17f-47dd-8c69-94e66628fb9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:57.264504 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.264460    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location" pod "d334e0d0-a17f-47dd-8c69-94e66628fb9c" (UID: "d334e0d0-a17f-47dd-8c69-94e66628fb9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:57.266498 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.266462    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls" pod "d334e0d0-a17f-47dd-8c69-94e66628fb9c" (UID: "d334e0d0-a17f-47dd-8c69-94e66628fb9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:57.266715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.266689    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6" pod "d334e0d0-a17f-47dd-8c69-94e66628fb9c" (UID: "d334e0d0-a17f-47dd-8c69-94e66628fb9c"). InnerVolumeSpecName "kube-api-access-5k6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:57.365190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.365108    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6q6\" (UniqueName: \"kubernetes.io/projected/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-api-access-5k6q6\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:57.365190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.365141    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d334e0d0-a17f-47dd-8c69-94e66628fb9c-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:57.365190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.365153    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:57.365190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.365162    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d334e0d0-a17f-47dd-8c69-94e66628fb9c-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:57.365190 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.365171    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d334e0d0-a17f-47dd-8c69-94e66628fb9c-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:57.883797 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.883755    4942 generic.go:356] "Generic (PLEG): container finished" podID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerID="5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b" exitCode=0
Oct 30 19:21:57.884242 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.883820    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"
Oct 30 19:21:57.884242 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.883811    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerDied","Data":"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"}
Oct 30 19:21:57.884242 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.883951    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt" event={"ID":"d334e0d0-a17f-47dd-8c69-94e66628fb9c","Type":"ContainerDied","Data":"be3d6096e4f1e9a90b4bb66d4c82b0ae0b043d6df4a5964993a40e04db63e1b4"}
Oct 30 19:21:57.884242 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.883981    4942 scope.go:122] "RemoveContainer" containerID="14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"
Oct 30 19:21:57.892402 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.892383    4942 scope.go:122] "RemoveContainer" containerID="5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"
Oct 30 19:21:57.906327 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.904406    4942 scope.go:122] "RemoveContainer" containerID="67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"
Oct 30 19:21:57.906327 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.906113    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:21:57.910889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.910867    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-runtime-predictor-6fcb44bc9f-n6bwt"]
Oct 30 19:21:57.914545 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.914530    4942 scope.go:122] "RemoveContainer" containerID="14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"
Oct 30 19:21:57.914854 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:21:57.914834    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b\": container with ID starting with 14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b not found: ID does not exist" containerID="14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"
Oct 30 19:21:57.914908 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.914863    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b"} err="failed to get container status \"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b\": rpc error: code = NotFound desc = could not find container \"14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b\": container with ID starting with 14a32dab76e59f3dd1c5ef22b4add669d266ab004d5c57f9ed6b28198a6ec71b not found: ID does not exist"
Oct 30 19:21:57.914908 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.914881    4942 scope.go:122] "RemoveContainer" containerID="5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"
Oct 30 19:21:57.915157 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:21:57.915132    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b\": container with ID starting with 5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b not found: ID does not exist" containerID="5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"
Oct 30 19:21:57.915236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.915167    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b"} err="failed to get container status \"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b\": rpc error: code = NotFound desc = could not find container \"5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b\": container with ID starting with 5cec649c5afb262d698538de77808bf601aa4b283a129853eb0381ddf9fb6f2b not found: ID does not exist"
Oct 30 19:21:57.915236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.915188    4942 scope.go:122] "RemoveContainer" containerID="67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"
Oct 30 19:21:57.915457 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:21:57.915441    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97\": container with ID starting with 67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97 not found: ID does not exist" containerID="67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"
Oct 30 19:21:57.915496 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:57.915463    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97"} err="failed to get container status \"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97\": rpc error: code = NotFound desc = could not find container \"67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97\": container with ID starting with 67691dfe6f2c64366740ff7f9ced6b7a061f92c3e176476f85af4a1c00a4cd97 not found: ID does not exist"
Oct 30 19:21:58.858299 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858264    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:21:58.858637 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858624    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container"
Oct 30 19:21:58.858691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858638    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container"
Oct 30 19:21:58.858691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858650    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kube-rbac-proxy"
Oct 30 19:21:58.858691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858655    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kube-rbac-proxy"
Oct 30 19:21:58.858691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858673    4942 cpu_manager.go:414] "containerMap: removing container" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="storage-initializer"
Oct 30 19:21:58.858691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858680    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="storage-initializer"
Oct 30 19:21:58.858843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858737    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kserve-container"
Oct 30 19:21:58.858843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.858746    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" containerName="kube-rbac-proxy"
Oct 30 19:21:58.863571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.863550    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.870380 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.870289    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:21:58.870469 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.870387    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:21:58.870469 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.870451    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:58.870469 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.870387    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 30 19:21:58.870580 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.870479    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:58.872091 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.872072    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:21:58.872200 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.872112    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:21:58.874345 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.874320    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:21:58.877417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.877397    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.877515 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.877445    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.877586 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.877545    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.877637 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.877584    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.877728 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.877705    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwhg\" (UniqueName: \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.978352 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978313    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwhg\" (UniqueName: \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.978837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978363    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.978837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978401    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.978837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978427    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.978837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978457    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.979115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.978859    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.979215 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.979191    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.979317 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.979294    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.980955 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.980937    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:58.992000 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:58.991970    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwhg\" (UniqueName: \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\") pod \"isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:59.181833 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:59.181796    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:21:59.462740 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:59.462668    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:21:59.465823 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:21:59.465794    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102d67ac_2164_4af1_b669_19b214933559.slice/crio-98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02 WatchSource:0}: Error finding container 98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02: Status 404 returned error can't find the container with id 98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02
Oct 30 19:21:59.541690 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:59.541649    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d334e0d0-a17f-47dd-8c69-94e66628fb9c" path="/var/lib/kubelet/pods/d334e0d0-a17f-47dd-8c69-94e66628fb9c/volumes"
Oct 30 19:21:59.894814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:59.894716    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerStarted","Data":"e1ec5e708afecb1585754a9a06471f5fc789ccc5760c9947bed6cca534e84336"}
Oct 30 19:21:59.894814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:21:59.894777    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerStarted","Data":"98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02"}
Oct 30 19:22:01.542066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.542038    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:22:01.742593 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.742558    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:22:01.742821 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.742792    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container" containerID="cri-o://a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc" gracePeriod=30
Oct 30 19:22:01.742885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.742846    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kube-rbac-proxy" containerID="cri-o://f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb" gracePeriod=30
Oct 30 19:22:01.743944 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.743922    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:22:01.905178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.905148    4942 generic.go:356] "Generic (PLEG): container finished" podID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerID="f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb" exitCode=0
Oct 30 19:22:01.905345 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:01.905214    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerDied","Data":"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"}
Oct 30 19:22:02.520272 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:02.520234    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 19:22:03.915437 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:03.915354    4942 generic.go:356] "Generic (PLEG): container finished" podID="102d67ac-2164-4af1-b669-19b214933559" containerID="e1ec5e708afecb1585754a9a06471f5fc789ccc5760c9947bed6cca534e84336" exitCode=0
Oct 30 19:22:03.915437 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:03.915424    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerDied","Data":"e1ec5e708afecb1585754a9a06471f5fc789ccc5760c9947bed6cca534e84336"}
Oct 30 19:22:04.924855 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:04.924815    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerStarted","Data":"835c8dbcda6de3c8a472ef3d2114fd89276f420b7c0df8ca9d6c6ccbc383b519"}
Oct 30 19:22:04.925344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:04.924880    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerStarted","Data":"a19566a1056ccc3da94b722d819025a3da43154370fac55e7ec1253ca43ce0ac"}
Oct 30 19:22:04.925344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:04.925078    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:22:04.925344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:04.925125    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:22:04.952353 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:04.952292    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podStartSLOduration=2.988506804 podStartE2EDuration="6.952276579s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.963769775s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:04.95147466 +0000 UTC m=+2711.529570365" watchObservedRunningTime="2026-10-30 19:22:04.952276579 +0000 UTC m=+2711.530372271"
Oct 30 19:22:07.095553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.095527    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:22:07.142588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.142504    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\") pod \"cef54acf-53ae-4b01-a802-9f74283c1c43\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") "
Oct 30 19:22:07.142588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.142545    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\") pod \"cef54acf-53ae-4b01-a802-9f74283c1c43\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") "
Oct 30 19:22:07.142588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.142567    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\" (UniqueName: \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\") pod \"cef54acf-53ae-4b01-a802-9f74283c1c43\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") "
Oct 30 19:22:07.142937 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.142608    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\") pod \"cef54acf-53ae-4b01-a802-9f74283c1c43\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") "
Oct 30 19:22:07.142937 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.142646    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\") pod \"cef54acf-53ae-4b01-a802-9f74283c1c43\" (UID: \"cef54acf-53ae-4b01-a802-9f74283c1c43\") "
Oct 30 19:22:07.143082 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.143049    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config" pod "cef54acf-53ae-4b01-a802-9f74283c1c43" (UID: "cef54acf-53ae-4b01-a802-9f74283c1c43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:07.143082 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.143064    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location" pod "cef54acf-53ae-4b01-a802-9f74283c1c43" (UID: "cef54acf-53ae-4b01-a802-9f74283c1c43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:07.143212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.143178    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert" pod "cef54acf-53ae-4b01-a802-9f74283c1c43" (UID: "cef54acf-53ae-4b01-a802-9f74283c1c43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:07.145193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.145162    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9" pod "cef54acf-53ae-4b01-a802-9f74283c1c43" (UID: "cef54acf-53ae-4b01-a802-9f74283c1c43"). InnerVolumeSpecName "kube-api-access-q2dt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:07.145879 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.145854    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls" pod "cef54acf-53ae-4b01-a802-9f74283c1c43" (UID: "cef54acf-53ae-4b01-a802-9f74283c1c43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:07.244389 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.244341    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cef54acf-53ae-4b01-a802-9f74283c1c43-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:07.244389 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.244386    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:07.244610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.244407    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dt9\" (UniqueName: \"kubernetes.io/projected/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-api-access-q2dt9\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:07.244610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.244424    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cef54acf-53ae-4b01-a802-9f74283c1c43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:07.244610 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.244446    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef54acf-53ae-4b01-a802-9f74283c1c43-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:07.940816 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.940782    4942 generic.go:356] "Generic (PLEG): container finished" podID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerID="a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc" exitCode=0
Oct 30 19:22:07.941002 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.940820    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerDied","Data":"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"}
Oct 30 19:22:07.941002 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.940842    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"
Oct 30 19:22:07.941002 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.940872    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f" event={"ID":"cef54acf-53ae-4b01-a802-9f74283c1c43","Type":"ContainerDied","Data":"ecd45435a1b0bf38af56f164a617650895f30e94c301dc5e64cabe0175129d7d"}
Oct 30 19:22:07.941002 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.940898    4942 scope.go:122] "RemoveContainer" containerID="f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"
Oct 30 19:22:07.949359 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.949334    4942 scope.go:122] "RemoveContainer" containerID="a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"
Oct 30 19:22:07.957657 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.957526    4942 scope.go:122] "RemoveContainer" containerID="9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"
Oct 30 19:22:07.964958 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.964941    4942 scope.go:122] "RemoveContainer" containerID="f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"
Oct 30 19:22:07.965244 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:22:07.965221    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb\": container with ID starting with f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb not found: ID does not exist" containerID="f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"
Oct 30 19:22:07.965302 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.965263    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb"} err="failed to get container status \"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb\": rpc error: code = NotFound desc = could not find container \"f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb\": container with ID starting with f66580c285de7def611eff8e09db504cecbc92d04113150b72cbbd088980bafb not found: ID does not exist"
Oct 30 19:22:07.965302 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.965280    4942 scope.go:122] "RemoveContainer" containerID="a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"
Oct 30 19:22:07.965550 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:22:07.965530    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc\": container with ID starting with a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc not found: ID does not exist" containerID="a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"
Oct 30 19:22:07.965614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.965561    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc"} err="failed to get container status \"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc\": rpc error: code = NotFound desc = could not find container \"a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc\": container with ID starting with a5bdbd96541079aec8971a8764b9b012db97f63959947063fffa0c40f4f02ddc not found: ID does not exist"
Oct 30 19:22:07.965614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.965596    4942 scope.go:122] "RemoveContainer" containerID="9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"
Oct 30 19:22:07.965850 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:22:07.965830    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe\": container with ID starting with 9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe not found: ID does not exist" containerID="9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"
Oct 30 19:22:07.965887 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.965857    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe"} err="failed to get container status \"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe\": rpc error: code = NotFound desc = could not find container \"9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe\": container with ID starting with 9c1111ad3eccf8bdd9013cf73e0d527d00294433d8858e102a4149d8e4972ffe not found: ID does not exist"
Oct 30 19:22:07.969513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.969491    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:22:07.977641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:07.977621    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-predictor-5d456c5878-pcn7f"]
Oct 30 19:22:09.543121 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:09.543081    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" path="/var/lib/kubelet/pods/cef54acf-53ae-4b01-a802-9f74283c1c43/volumes"
Oct 30 19:22:10.936054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:10.936003    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:22:26.727333 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:22:26.727294    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:22:40.945741 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:40.945635    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:22:50.938772 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:50.938737    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:22:56.723251 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:22:56.723213    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:22:59.168990 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:59.168952    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:22:59.169433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:59.169304    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container" containerID="cri-o://a19566a1056ccc3da94b722d819025a3da43154370fac55e7ec1253ca43ce0ac" gracePeriod=30
Oct 30 19:22:59.169433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:22:59.169341    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy" containerID="cri-o://835c8dbcda6de3c8a472ef3d2114fd89276f420b7c0df8ca9d6c6ccbc383b519" gracePeriod=30
Oct 30 19:23:00.188251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:00.188215    4942 generic.go:356] "Generic (PLEG): container finished" podID="102d67ac-2164-4af1-b669-19b214933559" containerID="835c8dbcda6de3c8a472ef3d2114fd89276f420b7c0df8ca9d6c6ccbc383b519" exitCode=0
Oct 30 19:23:00.188639 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:00.188283    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerDied","Data":"835c8dbcda6de3c8a472ef3d2114fd89276f420b7c0df8ca9d6c6ccbc383b519"}
Oct 30 19:23:00.930520 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:00.930481    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Oct 30 19:23:01.936971 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:01.936922    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.68:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:05.930077 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:05.930032    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Oct 30 19:23:06.215076 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.215048    4942 generic.go:356] "Generic (PLEG): container finished" podID="102d67ac-2164-4af1-b669-19b214933559" containerID="a19566a1056ccc3da94b722d819025a3da43154370fac55e7ec1253ca43ce0ac" exitCode=0
Oct 30 19:23:06.215199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.215118    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerDied","Data":"a19566a1056ccc3da94b722d819025a3da43154370fac55e7ec1253ca43ce0ac"}
Oct 30 19:23:06.215199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.215174    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2" event={"ID":"102d67ac-2164-4af1-b669-19b214933559","Type":"ContainerDied","Data":"98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02"}
Oct 30 19:23:06.215199 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.215195    4942 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98dda01e02480b4fbb08cbaa74de317ebf9c4d28a40193a2f265fac54fe9ac02"
Oct 30 19:23:06.216134 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.216116    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:23:06.249497 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.249477    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\") pod \"102d67ac-2164-4af1-b669-19b214933559\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") "
Oct 30 19:23:06.249607 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.249508    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\") pod \"102d67ac-2164-4af1-b669-19b214933559\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") "
Oct 30 19:23:06.249607 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.249541    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\" (UniqueName: \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\") pod \"102d67ac-2164-4af1-b669-19b214933559\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") "
Oct 30 19:23:06.249861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.249838    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert" pod "102d67ac-2164-4af1-b669-19b214933559" (UID: "102d67ac-2164-4af1-b669-19b214933559"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:06.249898 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.249882    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config" pod "102d67ac-2164-4af1-b669-19b214933559" (UID: "102d67ac-2164-4af1-b669-19b214933559"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:06.251513 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.251486    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls" pod "102d67ac-2164-4af1-b669-19b214933559" (UID: "102d67ac-2164-4af1-b669-19b214933559"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:06.350306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350278    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\" (UniqueName: \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\") pod \"102d67ac-2164-4af1-b669-19b214933559\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") "
Oct 30 19:23:06.350477 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350317    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\") pod \"102d67ac-2164-4af1-b669-19b214933559\" (UID: \"102d67ac-2164-4af1-b669-19b214933559\") "
Oct 30 19:23:06.350477 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350442    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/102d67ac-2164-4af1-b669-19b214933559-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.350477 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350453    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.350477 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350463    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/102d67ac-2164-4af1-b669-19b214933559-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.350720 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.350696    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location" pod "102d67ac-2164-4af1-b669-19b214933559" (UID: "102d67ac-2164-4af1-b669-19b214933559"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:06.352296 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.352273    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg" pod "102d67ac-2164-4af1-b669-19b214933559" (UID: "102d67ac-2164-4af1-b669-19b214933559"). InnerVolumeSpecName "kube-api-access-2nwhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:06.451436 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.451360    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwhg\" (UniqueName: \"kubernetes.io/projected/102d67ac-2164-4af1-b669-19b214933559-kube-api-access-2nwhg\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.451436 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:06.451388    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/102d67ac-2164-4af1-b669-19b214933559-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.219206 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:07.219175    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"
Oct 30 19:23:07.244352 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:07.244321    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:23:07.249590 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:07.249568    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-runtime-predictor-fd6fdfb44-htxl2"]
Oct 30 19:23:07.542303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:07.542227    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102d67ac-2164-4af1-b669-19b214933559" path="/var/lib/kubelet/pods/102d67ac-2164-4af1-b669-19b214933559/volumes"
Oct 30 19:23:08.735954 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.735918    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736300    4942 cpu_manager.go:414] "containerMap: removing container" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736313    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736327    4942 cpu_manager.go:414] "containerMap: removing container" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736332    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736339    4942 cpu_manager.go:414] "containerMap: removing container" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="storage-initializer"
Oct 30 19:23:08.736369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736344    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="storage-initializer"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736383    4942 cpu_manager.go:414] "containerMap: removing container" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736388    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736402    4942 cpu_manager.go:414] "containerMap: removing container" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736406    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736413    4942 cpu_manager.go:414] "containerMap: removing container" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="storage-initializer"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736418    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="storage-initializer"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736481    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736491    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kube-rbac-proxy"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736498    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef54acf-53ae-4b01-a802-9f74283c1c43" containerName="kserve-container"
Oct 30 19:23:08.736592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.736506    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="102d67ac-2164-4af1-b669-19b214933559" containerName="kserve-container"
Oct 30 19:23:08.741093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.741074    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.751723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.751701    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 30 19:23:08.757114 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.757066    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:23:08.757241 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.757137    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:23:08.757241 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.757159    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:23:08.757385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.757265    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:23:08.758268 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.758249    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:23:08.758484 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.758467    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:23:08.762882 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.762865    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:23:08.766480 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.766461    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.766539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.766490    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjgss\" (UniqueName: \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.766539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.766520    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.766633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.766609    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.766713 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.766698    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.867906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.867873    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.867906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.867904    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.867925    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.867981    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.867999    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjgss\" (UniqueName: \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868343 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.868322    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868606 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.868582    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.868683 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.868653    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.870414 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.870396    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:08.886460 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:08.886432    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjgss\" (UniqueName: \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\") pod \"isvc-xgboost-v2-predictor-8d8b7947-jhkdr\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") " pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:09.058524 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:09.058449    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:09.338499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:09.338470    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:23:09.340570 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:23:09.340542    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78083faa_fee1_499e_983b_76f5afc80c11.slice/crio-9e6d06756442cde19797d25e686e8a8898e033608f9349afba762b12311864ff WatchSource:0}: Error finding container 9e6d06756442cde19797d25e686e8a8898e033608f9349afba762b12311864ff: Status 404 returned error can't find the container with id 9e6d06756442cde19797d25e686e8a8898e033608f9349afba762b12311864ff
Oct 30 19:23:10.234359 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:10.234320    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerStarted","Data":"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"}
Oct 30 19:23:10.234754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:10.234386    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerStarted","Data":"9e6d06756442cde19797d25e686e8a8898e033608f9349afba762b12311864ff"}
Oct 30 19:23:14.252332 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:14.252298    4942 generic.go:356] "Generic (PLEG): container finished" podID="78083faa-fee1-499e-983b-76f5afc80c11" containerID="d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a" exitCode=0
Oct 30 19:23:14.252759 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:14.252356    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerDied","Data":"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"}
Oct 30 19:23:15.257960 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.257929    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerStarted","Data":"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"}
Oct 30 19:23:15.258398 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.257989    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerStarted","Data":"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"}
Oct 30 19:23:15.258398 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.258301    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:15.258398 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.258348    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:15.259793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.259764    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:15.287358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:15.287302    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podStartSLOduration=3.337671044 podStartE2EDuration="7.287284909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.949613865s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:15.287268402 +0000 UTC m=+2781.865364093" watchObservedRunningTime="2026-10-30 19:23:15.287284909 +0000 UTC m=+2781.865380596"
Oct 30 19:23:16.263397 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:16.263353    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:21.267606 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:21.267574    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:23:21.268059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:21.268035    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:26.724896 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:23:26.724861    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:23:31.268450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:31.268398    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:40.682081 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:40.682047    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:23:40.682562 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:40.682457    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" containerID="cri-o://adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c" gracePeriod=30
Oct 30 19:23:40.682847 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:40.682719    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kube-rbac-proxy" containerID="cri-o://e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901" gracePeriod=30
Oct 30 19:23:41.268991 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:41.268955    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:41.379255 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:41.379218    4942 generic.go:356] "Generic (PLEG): container finished" podID="a1934637-0051-4e8b-b081-4fcd6955e993" containerID="e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901" exitCode=0
Oct 30 19:23:41.379418 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:41.379289    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerDied","Data":"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"}
Oct 30 19:23:41.537515 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:41.537414    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 19:23:43.525265 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:43.525224    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Oct 30 19:23:45.117846 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.117822    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:23:45.264374 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264288    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\") pod \"a1934637-0051-4e8b-b081-4fcd6955e993\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") "
Oct 30 19:23:45.264374 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264329    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\" (UniqueName: \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\") pod \"a1934637-0051-4e8b-b081-4fcd6955e993\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") "
Oct 30 19:23:45.264620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264388    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") pod \"a1934637-0051-4e8b-b081-4fcd6955e993\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") "
Oct 30 19:23:45.264620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264477    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\") pod \"a1934637-0051-4e8b-b081-4fcd6955e993\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") "
Oct 30 19:23:45.264620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264528    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\") pod \"a1934637-0051-4e8b-b081-4fcd6955e993\" (UID: \"a1934637-0051-4e8b-b081-4fcd6955e993\") "
Oct 30 19:23:45.264763 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264684    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config" pod "a1934637-0051-4e8b-b081-4fcd6955e993" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:45.264840 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264807    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location" pod "a1934637-0051-4e8b-b081-4fcd6955e993" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.264903 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264878    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert" pod "a1934637-0051-4e8b-b081-4fcd6955e993" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:45.264955 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.264901    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.266496 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.266476    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls" pod "a1934637-0051-4e8b-b081-4fcd6955e993" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:45.266624 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.266608    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk" pod "a1934637-0051-4e8b-b081-4fcd6955e993" (UID: "a1934637-0051-4e8b-b081-4fcd6955e993"). InnerVolumeSpecName "kube-api-access-hcrgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:45.365683 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.365651    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1934637-0051-4e8b-b081-4fcd6955e993-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.365683 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.365679    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1934637-0051-4e8b-b081-4fcd6955e993-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.365683 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.365689    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrgk\" (UniqueName: \"kubernetes.io/projected/a1934637-0051-4e8b-b081-4fcd6955e993-kube-api-access-hcrgk\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.365899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.365698    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1934637-0051-4e8b-b081-4fcd6955e993-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.397103 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.397074    4942 generic.go:356] "Generic (PLEG): container finished" podID="a1934637-0051-4e8b-b081-4fcd6955e993" containerID="adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c" exitCode=0
Oct 30 19:23:45.397235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.397130    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerDied","Data":"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"}
Oct 30 19:23:45.397235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.397135    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"
Oct 30 19:23:45.397235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.397179    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml" event={"ID":"a1934637-0051-4e8b-b081-4fcd6955e993","Type":"ContainerDied","Data":"62cbaa5e8491a91eb6d2e7d99d040c8d993edeaa76589280744ed547a8bdd0ad"}
Oct 30 19:23:45.397235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.397206    4942 scope.go:122] "RemoveContainer" containerID="e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"
Oct 30 19:23:45.406913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.406892    4942 scope.go:122] "RemoveContainer" containerID="adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"
Oct 30 19:23:45.414729 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.414711    4942 scope.go:122] "RemoveContainer" containerID="2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"
Oct 30 19:23:45.421425 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.421404    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:23:45.422261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.422248    4942 scope.go:122] "RemoveContainer" containerID="e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"
Oct 30 19:23:45.422564 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:23:45.422544    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901\": container with ID starting with e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901 not found: ID does not exist" containerID="e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"
Oct 30 19:23:45.422637 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.422574    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901"} err="failed to get container status \"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901\": rpc error: code = NotFound desc = could not find container \"e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901\": container with ID starting with e744cdc94a78e3a090eeb9b978f9ada478b89bf0c9356d7eb9614deb55fb2901 not found: ID does not exist"
Oct 30 19:23:45.422637 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.422591    4942 scope.go:122] "RemoveContainer" containerID="adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"
Oct 30 19:23:45.422853 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:23:45.422833    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c\": container with ID starting with adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c not found: ID does not exist" containerID="adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"
Oct 30 19:23:45.422916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.422862    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c"} err="failed to get container status \"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c\": rpc error: code = NotFound desc = could not find container \"adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c\": container with ID starting with adfee5c38fec1457ef8753c02f49b8a89a46d1b1a309351e1d37a0b0dbf7992c not found: ID does not exist"
Oct 30 19:23:45.422916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.422887    4942 scope.go:122] "RemoveContainer" containerID="2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"
Oct 30 19:23:45.423133 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:23:45.423117    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb\": container with ID starting with 2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb not found: ID does not exist" containerID="2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"
Oct 30 19:23:45.423188 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.423137    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb"} err="failed to get container status \"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb\": rpc error: code = NotFound desc = could not find container \"2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb\": container with ID starting with 2ac5dc04609c9cd9a5e424fb12efef770f7798f990abc4a0db8045b0db8fcbeb not found: ID does not exist"
Oct 30 19:23:45.428069 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.428048    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-promote-v2-predictor-7565fb54fc-t8tml"]
Oct 30 19:23:45.542377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:45.542303    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" path="/var/lib/kubelet/pods/a1934637-0051-4e8b-b081-4fcd6955e993/volumes"
Oct 30 19:23:46.251690 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.251653    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252042    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kube-rbac-proxy"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252054    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kube-rbac-proxy"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252066    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="storage-initializer"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252071    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="storage-initializer"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252090    4942 cpu_manager.go:414] "containerMap: removing container" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252098    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252154    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kserve-container"
Oct 30 19:23:46.252218 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.252165    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1934637-0051-4e8b-b081-4fcd6955e993" containerName="kube-rbac-proxy"
Oct 30 19:23:46.256852 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.256831    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.265797 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.265777    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 30 19:23:46.266078 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.265989    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 19:23:46.266078 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.266032    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:23:46.266078 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.266004    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:23:46.266078 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.265998    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:23:46.266078 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.266078    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:23:46.266299 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.266143    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:23:46.272889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.272867    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:23:46.373309 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.373276    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.373462 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.373327    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.373462 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.373409    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvbt\" (UniqueName: \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.373536 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.373464    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.373570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.373532    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.410149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.410119    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:23:46.413855 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.413838    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.425756 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.425734    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:23:46.465629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.465600    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:23:46.474074 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474050    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474215 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474099    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474215 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474207    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvbt\" (UniqueName: \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474337 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474246    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474337 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474276    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474687    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.474981 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474951    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.475127 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.474956    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.476641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.476618    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.487540 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.487515    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvbt\" (UniqueName: \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\") pod \"isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.573836 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.573759    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:46.575020 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.574983    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.575141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.575048    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.575215 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.575196    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.575281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.575230    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cngj\" (UniqueName: \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.575281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.575267    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.676586 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.676545    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.676586 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.676584    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.676796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.676652    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.676796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.676672    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cngj\" (UniqueName: \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.676796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.676705    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.677201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.677162    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.677567 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.677545    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.677621 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.677545    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.679102 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.679084    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.690777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.690753    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cngj\" (UniqueName: \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\") pod \"isvc-canary-rollback-predictor-777dfc6864-6gnrl\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.732318 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.732294    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:46.857089 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:46.856995    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:23:46.860814 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:23:46.860787    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d712afd_a748_4492_955c_2fb102fc7549.slice/crio-d4ca6fe4ae2eb7ace496d70e683dc7604cfea4b953734938c558ba5fc2b2765b WatchSource:0}: Error finding container d4ca6fe4ae2eb7ace496d70e683dc7604cfea4b953734938c558ba5fc2b2765b: Status 404 returned error can't find the container with id d4ca6fe4ae2eb7ace496d70e683dc7604cfea4b953734938c558ba5fc2b2765b
Oct 30 19:23:47.012506 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:47.012474    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:23:47.016063 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:23:47.016036    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bde192b_c7bd_4b75_93fd_49572408ae8e.slice/crio-fd463e05c547781991dc72c408d109b72d6ba171d7875729c3daa6da7fcd2bf4 WatchSource:0}: Error finding container fd463e05c547781991dc72c408d109b72d6ba171d7875729c3daa6da7fcd2bf4: Status 404 returned error can't find the container with id fd463e05c547781991dc72c408d109b72d6ba171d7875729c3daa6da7fcd2bf4
Oct 30 19:23:47.410435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:47.410400    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerStarted","Data":"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"}
Oct 30 19:23:47.410902 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:47.410471    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerStarted","Data":"d4ca6fe4ae2eb7ace496d70e683dc7604cfea4b953734938c558ba5fc2b2765b"}
Oct 30 19:23:47.412252 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:47.412220    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerStarted","Data":"d5a9859f28bf78e2f4d09042aca219fd5a69263a5ce7659377275b50709d38be"}
Oct 30 19:23:47.412363 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:47.412291    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerStarted","Data":"fd463e05c547781991dc72c408d109b72d6ba171d7875729c3daa6da7fcd2bf4"}
Oct 30 19:23:51.268415 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:51.268378    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:23:51.435362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:51.435324    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerID="d5a9859f28bf78e2f4d09042aca219fd5a69263a5ce7659377275b50709d38be" exitCode=0
Oct 30 19:23:51.435528 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:51.435394    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerDied","Data":"d5a9859f28bf78e2f4d09042aca219fd5a69263a5ce7659377275b50709d38be"}
Oct 30 19:23:51.437152 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:51.437081    4942 generic.go:356] "Generic (PLEG): container finished" podID="9d712afd-a748-4492-955c-2fb102fc7549" containerID="4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7" exitCode=0
Oct 30 19:23:51.437152 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:51.437138    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerDied","Data":"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"}
Oct 30 19:23:52.444193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.444156    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerStarted","Data":"1e4a84b5e0acd1fe495e14e503f4c876649a45bfad099e261f448dd15819dc39"}
Oct 30 19:23:52.444611 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.444237    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerStarted","Data":"28710365e4687b2afd6424c050e16ebc0463f80027e71b36b0c305568d53229b"}
Oct 30 19:23:52.444793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.444760    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:52.444988 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.444967    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:52.446317 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.446282    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:23:52.446829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.446809    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerStarted","Data":"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"}
Oct 30 19:23:52.446918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.446854    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerStarted","Data":"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"}
Oct 30 19:23:52.447135 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.447120    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:52.447210 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.447165    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:52.448194 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.448170    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:23:52.472118 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.472072    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podStartSLOduration=2.77678073 podStartE2EDuration="6.472059958s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695279228s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:52.46969006 +0000 UTC m=+2819.047785762" watchObservedRunningTime="2026-10-30 19:23:52.472059958 +0000 UTC m=+2819.050155651"
Oct 30 19:23:52.497676 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:52.497631    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podStartSLOduration=2.763351507 podStartE2EDuration="6.497619095s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.734267588s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:52.496267353 +0000 UTC m=+2819.074363042" watchObservedRunningTime="2026-10-30 19:23:52.497619095 +0000 UTC m=+2819.075714786"
Oct 30 19:23:53.452229 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:53.452180    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:23:53.452734 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:53.452325    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:23:54.455836 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:54.455799    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:23:56.723998 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:23:56.723959    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:23:58.457410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:58.457380    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:23:58.457884 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:58.457857    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:23:59.460966 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:59.460933    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:23:59.461499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:23:59.461471    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:01.268748 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:01.268712    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:24:08.458712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:08.458634    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:09.461488 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:09.461452    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:11.269025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:11.268968    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:24:18.458741 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:18.458701    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:19.461708 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:19.461669    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:21.268066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:21.268023    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:24:26.724075 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:24:26.723973    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:24:28.457913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:28.457871    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:29.461636 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:29.461594    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:31.268674 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:31.268639    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:24:38.458647 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:38.458602    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:39.461659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:39.461608    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:39.659162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:39.659126    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:24:39.659498 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:39.659447    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" containerID="cri-o://f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1" gracePeriod=30
Oct 30 19:24:39.659584 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:39.659519    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kube-rbac-proxy" containerID="cri-o://2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7" gracePeriod=30
Oct 30 19:24:40.667906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:40.667869    4942 generic.go:356] "Generic (PLEG): container finished" podID="78083faa-fee1-499e-983b-76f5afc80c11" containerID="2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7" exitCode=0
Oct 30 19:24:40.668341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:40.667940    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerDied","Data":"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"}
Oct 30 19:24:41.263841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:41.263799    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Oct 30 19:24:41.269164 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:41.269123    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 19:24:44.519142 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.519100    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:24:44.674671 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674562    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\") pod \"78083faa-fee1-499e-983b-76f5afc80c11\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") "
Oct 30 19:24:44.674671 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674624    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\" (UniqueName: \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\") pod \"78083faa-fee1-499e-983b-76f5afc80c11\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") "
Oct 30 19:24:44.674671 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674648    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\" (UniqueName: \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\") pod \"78083faa-fee1-499e-983b-76f5afc80c11\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") "
Oct 30 19:24:44.674951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674682    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\") pod \"78083faa-fee1-499e-983b-76f5afc80c11\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") "
Oct 30 19:24:44.674951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674921    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\") pod \"78083faa-fee1-499e-983b-76f5afc80c11\" (UID: \"78083faa-fee1-499e-983b-76f5afc80c11\") "
Oct 30 19:24:44.675079 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.674953    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location" pod "78083faa-fee1-499e-983b-76f5afc80c11" (UID: "78083faa-fee1-499e-983b-76f5afc80c11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:44.676986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.675462    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert" pod "78083faa-fee1-499e-983b-76f5afc80c11" (UID: "78083faa-fee1-499e-983b-76f5afc80c11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:44.676986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.675920    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:44.676986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.675933    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config" pod "78083faa-fee1-499e-983b-76f5afc80c11" (UID: "78083faa-fee1-499e-983b-76f5afc80c11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:44.676986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.675954    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78083faa-fee1-499e-983b-76f5afc80c11-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:44.683490 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.683426    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls" pod "78083faa-fee1-499e-983b-76f5afc80c11" (UID: "78083faa-fee1-499e-983b-76f5afc80c11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:44.685310 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.685264    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss" pod "78083faa-fee1-499e-983b-76f5afc80c11" (UID: "78083faa-fee1-499e-983b-76f5afc80c11"). InnerVolumeSpecName "kube-api-access-pjgss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:44.690426 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.690375    4942 generic.go:356] "Generic (PLEG): container finished" podID="78083faa-fee1-499e-983b-76f5afc80c11" containerID="f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1" exitCode=0
Oct 30 19:24:44.690609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.690451    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerDied","Data":"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"}
Oct 30 19:24:44.690609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.690559    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr" event={"ID":"78083faa-fee1-499e-983b-76f5afc80c11","Type":"ContainerDied","Data":"9e6d06756442cde19797d25e686e8a8898e033608f9349afba762b12311864ff"}
Oct 30 19:24:44.690609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.690577    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"
Oct 30 19:24:44.690609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.690595    4942 scope.go:122] "RemoveContainer" containerID="2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"
Oct 30 19:24:44.701043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.701000    4942 scope.go:122] "RemoveContainer" containerID="f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"
Oct 30 19:24:44.710207 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.710166    4942 scope.go:122] "RemoveContainer" containerID="d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"
Oct 30 19:24:44.718173 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.718132    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:24:44.720426 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.720392    4942 scope.go:122] "RemoveContainer" containerID="2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"
Oct 30 19:24:44.720916 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:24:44.720884    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7\": container with ID starting with 2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7 not found: ID does not exist" containerID="2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"
Oct 30 19:24:44.721268 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.721224    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7"} err="failed to get container status \"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7\": rpc error: code = NotFound desc = could not find container \"2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7\": container with ID starting with 2351e59661672fb6e7cb0ba441b3f5845c436dcb0c4fb65df32f248975456aa7 not found: ID does not exist"
Oct 30 19:24:44.721398 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.721268    4942 scope.go:122] "RemoveContainer" containerID="f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"
Oct 30 19:24:44.721646 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:24:44.721621    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1\": container with ID starting with f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1 not found: ID does not exist" containerID="f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"
Oct 30 19:24:44.721715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.721675    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1"} err="failed to get container status \"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1\": rpc error: code = NotFound desc = could not find container \"f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1\": container with ID starting with f4d85732b41875cbd04e517c83e42bb24e11b1a34a592f03a794e55271bb39f1 not found: ID does not exist"
Oct 30 19:24:44.721715 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.721694    4942 scope.go:122] "RemoveContainer" containerID="d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"
Oct 30 19:24:44.721969 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:24:44.721952    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a\": container with ID starting with d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a not found: ID does not exist" containerID="d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"
Oct 30 19:24:44.722057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.721972    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a"} err="failed to get container status \"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a\": rpc error: code = NotFound desc = could not find container \"d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a\": container with ID starting with d1b1446ed59dcfc5e4691eaa2f2a5f6af7db53e275be24fae5bfa4aeec5e9a7a not found: ID does not exist"
Oct 30 19:24:44.722752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.722726    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-xgboost-v2-predictor-8d8b7947-jhkdr"]
Oct 30 19:24:44.776468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.776420    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78083faa-fee1-499e-983b-76f5afc80c11-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:44.776468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.776454    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjgss\" (UniqueName: \"kubernetes.io/projected/78083faa-fee1-499e-983b-76f5afc80c11-kube-api-access-pjgss\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:44.776664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:44.776477    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78083faa-fee1-499e-983b-76f5afc80c11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:45.390499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390461    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:24:45.390845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390831    4942 cpu_manager.go:414] "containerMap: removing container" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container"
Oct 30 19:24:45.390907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390845    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container"
Oct 30 19:24:45.390907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390857    4942 cpu_manager.go:414] "containerMap: removing container" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="storage-initializer"
Oct 30 19:24:45.390907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390863    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="storage-initializer"
Oct 30 19:24:45.390907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390881    4942 cpu_manager.go:414] "containerMap: removing container" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kube-rbac-proxy"
Oct 30 19:24:45.390907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390887    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kube-rbac-proxy"
Oct 30 19:24:45.391105 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390946    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kserve-container"
Oct 30 19:24:45.391105 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.390958    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="78083faa-fee1-499e-983b-76f5afc80c11" containerName="kube-rbac-proxy"
Oct 30 19:24:45.395931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.395898    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.412120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.412087    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:24:45.414886 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.414854    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:24:45.415126 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.414913    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 30 19:24:45.415126 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.415093    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"storage-config\""
Oct 30 19:24:45.416590 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.416562    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 30 19:24:45.417166 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.417143    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:24:45.418273 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.418251    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:24:45.484756 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.484718    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.484939 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.484775    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.484939 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.484876    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqzb\" (UniqueName: \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.484939 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.484913    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.542798 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.542759    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78083faa-fee1-499e-983b-76f5afc80c11" path="/var/lib/kubelet/pods/78083faa-fee1-499e-983b-76f5afc80c11/volumes"
Oct 30 19:24:45.586063 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.585989    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqzb\" (UniqueName: \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.586283 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.586082    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.586283 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.586131    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.586283 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.586250    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.586712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.586681    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.587056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.587036    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.588654 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.588631    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.600246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.600203    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqzb\" (UniqueName: \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\") pod \"isvc-sklearn-s3-predictor-678d78cf79-wpvbr\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:45.714143 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:45.714092    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:46.002626 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:46.002399    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:24:46.005699 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:24:46.005666    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4b29db_e2f6_4701_a9ec_7334eaca0579.slice/crio-36f3e0f7e85e896f0c1bf8c5a86f1fdeb2098e27f6baee1f427bb4a6b5513b17 WatchSource:0}: Error finding container 36f3e0f7e85e896f0c1bf8c5a86f1fdeb2098e27f6baee1f427bb4a6b5513b17: Status 404 returned error can't find the container with id 36f3e0f7e85e896f0c1bf8c5a86f1fdeb2098e27f6baee1f427bb4a6b5513b17
Oct 30 19:24:46.703246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:46.703200    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerStarted","Data":"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"}
Oct 30 19:24:46.703699 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:46.703281    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerStarted","Data":"36f3e0f7e85e896f0c1bf8c5a86f1fdeb2098e27f6baee1f427bb4a6b5513b17"}
Oct 30 19:24:47.710539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:47.710493    4942 generic.go:356] "Generic (PLEG): container finished" podID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerID="0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940" exitCode=0
Oct 30 19:24:47.710992 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:47.710583    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerDied","Data":"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"}
Oct 30 19:24:48.458539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:48.458488    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:48.718936 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:48.718837    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerStarted","Data":"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"}
Oct 30 19:24:48.719425 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:48.719040    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:48.719425 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:48.719102    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerStarted","Data":"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"}
Oct 30 19:24:48.745324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:48.745260    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podStartSLOduration=2.442663192 podStartE2EDuration="3.745241123s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.302577931s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:48.743876593 +0000 UTC m=+2875.321972286" watchObservedRunningTime="2026-10-30 19:24:48.745241123 +0000 UTC m=+2875.323336815"
Oct 30 19:24:49.461608 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:49.461563    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:24:49.724287 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:49.724192    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:49.725817 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:49.725782    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:24:50.728912 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:50.728867    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:24:55.733588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:55.733541    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:24:55.734288 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:55.734252    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:24:56.730076 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:24:56.730036    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:24:58.457902 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:58.457861    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:24:59.461969 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:24:59.461918    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:25:05.734182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:05.734140    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:08.458845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:08.458794    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:25:09.461574 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:09.461527    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:25:11.537349 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:11.537275    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 19:25:15.734282 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:15.734234    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:18.458830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.458797    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:25:18.684018 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.683971    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:25:18.684253 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.684227    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container" containerID="cri-o://28710365e4687b2afd6424c050e16ebc0463f80027e71b36b0c305568d53229b" gracePeriod=30
Oct 30 19:25:18.684354 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.684300    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kube-rbac-proxy" containerID="cri-o://1e4a84b5e0acd1fe495e14e503f4c876649a45bfad099e261f448dd15819dc39" gracePeriod=30
Oct 30 19:25:18.685178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.685156    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:25:18.863035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.862985    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerID="1e4a84b5e0acd1fe495e14e503f4c876649a45bfad099e261f448dd15819dc39" exitCode=0
Oct 30 19:25:18.863203 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:18.863057    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerDied","Data":"1e4a84b5e0acd1fe495e14e503f4c876649a45bfad099e261f448dd15819dc39"}
Oct 30 19:25:19.456222 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:19.456181    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.71:8643/healthz\": dial tcp 10.132.0.71:8643: connect: connection refused"
Oct 30 19:25:23.888820 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.888785    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerID="28710365e4687b2afd6424c050e16ebc0463f80027e71b36b0c305568d53229b" exitCode=0
Oct 30 19:25:23.889213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.888863    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerDied","Data":"28710365e4687b2afd6424c050e16ebc0463f80027e71b36b0c305568d53229b"}
Oct 30 19:25:23.932048 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.932024    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:25:23.983999 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.983966    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\" (UniqueName: \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\") pod \"3bde192b-c7bd-4b75-93fd-49572408ae8e\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") "
Oct 30 19:25:23.984216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984027    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\") pod \"3bde192b-c7bd-4b75-93fd-49572408ae8e\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") "
Oct 30 19:25:23.984216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984096    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\") pod \"3bde192b-c7bd-4b75-93fd-49572408ae8e\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") "
Oct 30 19:25:23.984216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984186    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\") pod \"3bde192b-c7bd-4b75-93fd-49572408ae8e\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") "
Oct 30 19:25:23.984390 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984248    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\") pod \"3bde192b-c7bd-4b75-93fd-49572408ae8e\" (UID: \"3bde192b-c7bd-4b75-93fd-49572408ae8e\") "
Oct 30 19:25:23.984570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984542    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config" pod "3bde192b-c7bd-4b75-93fd-49572408ae8e" (UID: "3bde192b-c7bd-4b75-93fd-49572408ae8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:23.984713 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984572    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert" pod "3bde192b-c7bd-4b75-93fd-49572408ae8e" (UID: "3bde192b-c7bd-4b75-93fd-49572408ae8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:23.984713 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.984596    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location" pod "3bde192b-c7bd-4b75-93fd-49572408ae8e" (UID: "3bde192b-c7bd-4b75-93fd-49572408ae8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:23.986800 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.986772    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj" pod "3bde192b-c7bd-4b75-93fd-49572408ae8e" (UID: "3bde192b-c7bd-4b75-93fd-49572408ae8e"). InnerVolumeSpecName "kube-api-access-5cngj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:23.986800 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:23.986780    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls" pod "3bde192b-c7bd-4b75-93fd-49572408ae8e" (UID: "3bde192b-c7bd-4b75-93fd-49572408ae8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:24.085670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.085586    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:24.085670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.085617    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:24.085670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.085627    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bde192b-c7bd-4b75-93fd-49572408ae8e-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:24.085670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.085637    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cngj\" (UniqueName: \"kubernetes.io/projected/3bde192b-c7bd-4b75-93fd-49572408ae8e-kube-api-access-5cngj\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:24.085670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.085645    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bde192b-c7bd-4b75-93fd-49572408ae8e-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:24.895381 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.895355    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"
Oct 30 19:25:24.895381 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.895364    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl" event={"ID":"3bde192b-c7bd-4b75-93fd-49572408ae8e","Type":"ContainerDied","Data":"fd463e05c547781991dc72c408d109b72d6ba171d7875729c3daa6da7fcd2bf4"}
Oct 30 19:25:24.895937 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.895424    4942 scope.go:122] "RemoveContainer" containerID="1e4a84b5e0acd1fe495e14e503f4c876649a45bfad099e261f448dd15819dc39"
Oct 30 19:25:24.905247 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.905227    4942 scope.go:122] "RemoveContainer" containerID="28710365e4687b2afd6424c050e16ebc0463f80027e71b36b0c305568d53229b"
Oct 30 19:25:24.913025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.912988    4942 scope.go:122] "RemoveContainer" containerID="d5a9859f28bf78e2f4d09042aca219fd5a69263a5ce7659377275b50709d38be"
Oct 30 19:25:24.921140 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.921116    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:25:24.925515 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:24.925493    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6gnrl"]
Oct 30 19:25:25.542397 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:25.542366    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" path="/var/lib/kubelet/pods/3bde192b-c7bd-4b75-93fd-49572408ae8e/volumes"
Oct 30 19:25:25.734232 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:25.734195    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:26.422327 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.422275    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:25:26.422880 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.422633    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kube-rbac-proxy" containerID="cri-o://21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5" gracePeriod=30
Oct 30 19:25:26.422880 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.422834    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" containerID="cri-o://cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb" gracePeriod=30
Oct 30 19:25:26.492799 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.492761    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:25:26.493158 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493143    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container"
Oct 30 19:25:26.493212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493158    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container"
Oct 30 19:25:26.493212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493175    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kube-rbac-proxy"
Oct 30 19:25:26.493212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493180    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kube-rbac-proxy"
Oct 30 19:25:26.493212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493190    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="storage-initializer"
Oct 30 19:25:26.493212 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493205    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="storage-initializer"
Oct 30 19:25:26.493422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493271    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kserve-container"
Oct 30 19:25:26.493422 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.493279    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bde192b-c7bd-4b75-93fd-49572408ae8e" containerName="kube-rbac-proxy"
Oct 30 19:25:26.496434 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.496411    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.503649 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.503622    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:25:26.511410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.511377    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:25:26.605807 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.605752    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.606055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.605889    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.606055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.605912    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.606055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.605956    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.606200 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.606137    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.706573 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.706534    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.706791 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.706587    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.706791 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.706620    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.706791 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.706736    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.706976 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.706808    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.714291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.707603    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.714291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.707672    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.714291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.707883    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.714291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.710209    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.720344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.720294    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\") pod \"isvc-canary-rollback-predictor-777dfc6864-6d9hb\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") " pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.729372 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:25:26.729321    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:25:26.816091 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.816045    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:26.908900 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.908863    4942 generic.go:356] "Generic (PLEG): container finished" podID="9d712afd-a748-4492-955c-2fb102fc7549" containerID="21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5" exitCode=0
Oct 30 19:25:26.909079 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:26.908902    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerDied","Data":"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"}
Oct 30 19:25:27.097383 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:27.097356    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:25:27.099998 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:25:27.099965    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bb0f04_65a3_40ea_879c_0e8b3786b2e5.slice/crio-bcb570426a23d1cd892ec8ddc4cd917e5663dd6fcc4a052496e9b7ebd5abcecc WatchSource:0}: Error finding container bcb570426a23d1cd892ec8ddc4cd917e5663dd6fcc4a052496e9b7ebd5abcecc: Status 404 returned error can't find the container with id bcb570426a23d1cd892ec8ddc4cd917e5663dd6fcc4a052496e9b7ebd5abcecc
Oct 30 19:25:27.914341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:27.914302    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerStarted","Data":"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"}
Oct 30 19:25:27.914736 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:27.914374    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerStarted","Data":"bcb570426a23d1cd892ec8ddc4cd917e5663dd6fcc4a052496e9b7ebd5abcecc"}
Oct 30 19:25:28.453361 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:28.453316    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.70:8643/healthz\": dial tcp 10.132.0.70:8643: connect: connection refused"
Oct 30 19:25:28.458441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:28.458399    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 19:25:31.934712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:31.934672    4942 generic.go:356] "Generic (PLEG): container finished" podID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerID="8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed" exitCode=0
Oct 30 19:25:31.935216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:31.934749    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerDied","Data":"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"}
Oct 30 19:25:32.197324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.197224    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:25:32.252181 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252117    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\") pod \"9d712afd-a748-4492-955c-2fb102fc7549\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") "
Oct 30 19:25:32.252385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252228    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\") pod \"9d712afd-a748-4492-955c-2fb102fc7549\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") "
Oct 30 19:25:32.252385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252269    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\") pod \"9d712afd-a748-4492-955c-2fb102fc7549\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") "
Oct 30 19:25:32.252385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252312    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\") pod \"9d712afd-a748-4492-955c-2fb102fc7549\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") "
Oct 30 19:25:32.252385 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252354    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\" (UniqueName: \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\") pod \"9d712afd-a748-4492-955c-2fb102fc7549\" (UID: \"9d712afd-a748-4492-955c-2fb102fc7549\") "
Oct 30 19:25:32.252625 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252532    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location" pod "9d712afd-a748-4492-955c-2fb102fc7549" (UID: "9d712afd-a748-4492-955c-2fb102fc7549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:32.252731 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.252701    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d712afd-a748-4492-955c-2fb102fc7549-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:32.253178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.253145    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config" pod "9d712afd-a748-4492-955c-2fb102fc7549" (UID: "9d712afd-a748-4492-955c-2fb102fc7549"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:32.253314 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.253239    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert" pod "9d712afd-a748-4492-955c-2fb102fc7549" (UID: "9d712afd-a748-4492-955c-2fb102fc7549"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:32.255249 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.255208    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls" pod "9d712afd-a748-4492-955c-2fb102fc7549" (UID: "9d712afd-a748-4492-955c-2fb102fc7549"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:32.255249 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.255206    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt" pod "9d712afd-a748-4492-955c-2fb102fc7549" (UID: "9d712afd-a748-4492-955c-2fb102fc7549"). InnerVolumeSpecName "kube-api-access-vxvbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:32.353768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.353724    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:32.353768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.353761    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d712afd-a748-4492-955c-2fb102fc7549-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:32.353768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.353772    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d712afd-a748-4492-955c-2fb102fc7549-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:32.354030 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.353781    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvbt\" (UniqueName: \"kubernetes.io/projected/9d712afd-a748-4492-955c-2fb102fc7549-kube-api-access-vxvbt\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:32.942623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.942583    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerStarted","Data":"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"}
Oct 30 19:25:32.943133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.942667    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerStarted","Data":"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"}
Oct 30 19:25:32.943133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.942867    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:32.945258 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.945221    4942 generic.go:356] "Generic (PLEG): container finished" podID="9d712afd-a748-4492-955c-2fb102fc7549" containerID="cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb" exitCode=0
Oct 30 19:25:32.945400 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.945283    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerDied","Data":"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"}
Oct 30 19:25:32.945400 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.945290    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"
Oct 30 19:25:32.945400 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.945346    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k" event={"ID":"9d712afd-a748-4492-955c-2fb102fc7549","Type":"ContainerDied","Data":"d4ca6fe4ae2eb7ace496d70e683dc7604cfea4b953734938c558ba5fc2b2765b"}
Oct 30 19:25:32.945400 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.945386    4942 scope.go:122] "RemoveContainer" containerID="21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"
Oct 30 19:25:32.957043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.956993    4942 scope.go:122] "RemoveContainer" containerID="cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"
Oct 30 19:25:32.968087 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.968054    4942 scope.go:122] "RemoveContainer" containerID="4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"
Oct 30 19:25:32.972741 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.972679    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podStartSLOduration=3.134857154 podStartE2EDuration="6.972663798s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837806644s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.971387032 +0000 UTC m=+2919.549482723" watchObservedRunningTime="2026-10-30 19:25:32.972663798 +0000 UTC m=+2919.550759486"
Oct 30 19:25:32.978229 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.978204    4942 scope.go:122] "RemoveContainer" containerID="21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"
Oct 30 19:25:32.978646 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:25:32.978624    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5\": container with ID starting with 21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5 not found: ID does not exist" containerID="21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"
Oct 30 19:25:32.978738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.978652    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5"} err="failed to get container status \"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5\": rpc error: code = NotFound desc = could not find container \"21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5\": container with ID starting with 21259724ba45a0abb7197a1b552b4d8b902c69bb5ce15d98f35c1277a402f1d5 not found: ID does not exist"
Oct 30 19:25:32.978738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.978669    4942 scope.go:122] "RemoveContainer" containerID="cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"
Oct 30 19:25:32.978966 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:25:32.978944    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb\": container with ID starting with cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb not found: ID does not exist" containerID="cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"
Oct 30 19:25:32.979061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.978972    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb"} err="failed to get container status \"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb\": rpc error: code = NotFound desc = could not find container \"cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb\": container with ID starting with cd0f99fa6fb8dbb5e19bd1e00a063e73fe3660c23f045e4449b2a29ef2927acb not found: ID does not exist"
Oct 30 19:25:32.979061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.978992    4942 scope.go:122] "RemoveContainer" containerID="4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"
Oct 30 19:25:32.979379 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:25:32.979354    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7\": container with ID starting with 4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7 not found: ID does not exist" containerID="4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"
Oct 30 19:25:32.979451 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.979385    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7"} err="failed to get container status \"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7\": rpc error: code = NotFound desc = could not find container \"4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7\": container with ID starting with 4577d6bd90006446bf7131cc8814c2ecd8684322151f340ca4820df50feb66c7 not found: ID does not exist"
Oct 30 19:25:32.987679 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.987622    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:25:32.990530 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:32.990488    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-v2-predictor-5dd6549fc7-75w9k"]
Oct 30 19:25:33.542821 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:33.542778    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d712afd-a748-4492-955c-2fb102fc7549" path="/var/lib/kubelet/pods/9d712afd-a748-4492-955c-2fb102fc7549/volumes"
Oct 30 19:25:33.953193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:33.953153    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:33.954738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:33.954702    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:25:34.957440 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:34.957397    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:25:35.734301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:35.734244    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:39.961759 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:39.961727    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:25:39.962413 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:39.962384    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:25:45.734630 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:45.734586    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:49.962613 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:49.962570    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:25:55.734997 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:55.734957    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:25:56.724159 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:25:56.724117    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:25:59.962787 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:25:59.962749    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:05.734902 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:05.734859    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 19:26:09.962356 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:09.962316    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:15.735182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:15.735149    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:26:19.962673 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:19.962631    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:26.228432 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:26.228396    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:26:26.228930 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:26.228754    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container" containerID="cri-o://3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe" gracePeriod=30
Oct 30 19:26:26.228930 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:26.228819    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kube-rbac-proxy" containerID="cri-o://69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f" gracePeriod=30
Oct 30 19:26:26.725524 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:26:26.725487    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:26:27.180798 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:27.180766    4942 generic.go:356] "Generic (PLEG): container finished" podID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerID="69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f" exitCode=0
Oct 30 19:26:27.180956 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:27.180838    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerDied","Data":"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"}
Oct 30 19:26:29.962408 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:29.962371    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:30.729693 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:30.729648    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Oct 30 19:26:31.171167 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.171139    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:26:31.200303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.200262    4942 generic.go:356] "Generic (PLEG): container finished" podID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerID="3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe" exitCode=0
Oct 30 19:26:31.200482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.200304    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerDied","Data":"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"}
Oct 30 19:26:31.200482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.200339    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"
Oct 30 19:26:31.200482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.200378    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr" event={"ID":"ea4b29db-e2f6-4701-a9ec-7334eaca0579","Type":"ContainerDied","Data":"36f3e0f7e85e896f0c1bf8c5a86f1fdeb2098e27f6baee1f427bb4a6b5513b17"}
Oct 30 19:26:31.200482 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.200421    4942 scope.go:122] "RemoveContainer" containerID="69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"
Oct 30 19:26:31.213185 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.213163    4942 scope.go:122] "RemoveContainer" containerID="3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"
Oct 30 19:26:31.222784 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.222762    4942 scope.go:122] "RemoveContainer" containerID="0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"
Oct 30 19:26:31.231359 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.231339    4942 scope.go:122] "RemoveContainer" containerID="69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"
Oct 30 19:26:31.231658 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:26:31.231630    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f\": container with ID starting with 69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f not found: ID does not exist" containerID="69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"
Oct 30 19:26:31.231753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.231674    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f"} err="failed to get container status \"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f\": rpc error: code = NotFound desc = could not find container \"69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f\": container with ID starting with 69c2e4597b5e4969444bd43ad40054a7ff8cf2f6c2ef2f5e58bb54be8eb8274f not found: ID does not exist"
Oct 30 19:26:31.231753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.231701    4942 scope.go:122] "RemoveContainer" containerID="3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"
Oct 30 19:26:31.232001 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:26:31.231981    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe\": container with ID starting with 3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe not found: ID does not exist" containerID="3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"
Oct 30 19:26:31.232075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.232025    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe"} err="failed to get container status \"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe\": rpc error: code = NotFound desc = could not find container \"3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe\": container with ID starting with 3c0079b8e1446bb10f488269be74a87ece01aa2808627da9251b634f8be6e0fe not found: ID does not exist"
Oct 30 19:26:31.232075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.232044    4942 scope.go:122] "RemoveContainer" containerID="0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"
Oct 30 19:26:31.232299 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:26:31.232279    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940\": container with ID starting with 0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940 not found: ID does not exist" containerID="0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"
Oct 30 19:26:31.232338 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.232304    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940"} err="failed to get container status \"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940\": rpc error: code = NotFound desc = could not find container \"0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940\": container with ID starting with 0e09011554cabaf3777e6024cd4ec2b8693882b934d501f30fbf060074250940 not found: ID does not exist"
Oct 30 19:26:31.264841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.264809    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\") pod \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") "
Oct 30 19:26:31.265035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.264890    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\" (UniqueName: \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\") pod \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") "
Oct 30 19:26:31.265035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.264917    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\") pod \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") "
Oct 30 19:26:31.265035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.264943    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\") pod \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\" (UID: \"ea4b29db-e2f6-4701-a9ec-7334eaca0579\") "
Oct 30 19:26:31.265304 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.265280    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location" pod "ea4b29db-e2f6-4701-a9ec-7334eaca0579" (UID: "ea4b29db-e2f6-4701-a9ec-7334eaca0579"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:31.265368 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.265308    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config" pod "ea4b29db-e2f6-4701-a9ec-7334eaca0579" (UID: "ea4b29db-e2f6-4701-a9ec-7334eaca0579"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:31.266962 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.266943    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb" pod "ea4b29db-e2f6-4701-a9ec-7334eaca0579" (UID: "ea4b29db-e2f6-4701-a9ec-7334eaca0579"). InnerVolumeSpecName "kube-api-access-fcqzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:31.267033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.266970    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls" pod "ea4b29db-e2f6-4701-a9ec-7334eaca0579" (UID: "ea4b29db-e2f6-4701-a9ec-7334eaca0579"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:31.366057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.365941    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4b29db-e2f6-4701-a9ec-7334eaca0579-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:31.366057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.365975    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqzb\" (UniqueName: \"kubernetes.io/projected/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-api-access-fcqzb\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:31.366057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.365989    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:31.366057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.365999    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4b29db-e2f6-4701-a9ec-7334eaca0579-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:31.524837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.524803    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:26:31.527838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.527811    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-predictor-678d78cf79-wpvbr"]
Oct 30 19:26:31.542048 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:31.542020    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" path="/var/lib/kubelet/pods/ea4b29db-e2f6-4701-a9ec-7334eaca0579/volumes"
Oct 30 19:26:33.983117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983083    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983418    4942 cpu_manager.go:414] "containerMap: removing container" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kube-rbac-proxy"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983429    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kube-rbac-proxy"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983446    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="storage-initializer"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983451    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="storage-initializer"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983457    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kube-rbac-proxy"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983463    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kube-rbac-proxy"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983469    4942 cpu_manager.go:414] "containerMap: removing container" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983474    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983487    4942 cpu_manager.go:414] "containerMap: removing container" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983491    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container"
Oct 30 19:26:33.983493 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983499    4942 cpu_manager.go:414] "containerMap: removing container" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="storage-initializer"
Oct 30 19:26:33.983921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983504    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="storage-initializer"
Oct 30 19:26:33.983921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983551    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kube-rbac-proxy"
Oct 30 19:26:33.983921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983558    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kserve-container"
Oct 30 19:26:33.983921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983565    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4b29db-e2f6-4701-a9ec-7334eaca0579" containerName="kserve-container"
Oct 30 19:26:33.983921 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.983575    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d712afd-a748-4492-955c-2fb102fc7549" containerName="kube-rbac-proxy"
Oct 30 19:26:33.988381 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.988364    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:33.998448 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998427    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:26:33.998557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998493    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:26:33.998557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998506    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:26:33.998557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998521    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:26:33.998557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998481    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"storage-config\""
Oct 30 19:26:33.998702 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:33.998607    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 30 19:26:34.001494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.001475    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:26:34.012178 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.012145    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:26:34.086317 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.086286    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.086317 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.086321    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.086566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.086363    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdrv\" (UniqueName: \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.086566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.086484    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.086566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.086546    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187146 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187105    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187146 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187148    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187404 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187206    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdrv\" (UniqueName: \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187404 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187238    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187516 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187443    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187748    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187900 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187878    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.187965 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.187916    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.189724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.189706    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.200401 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.200374    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdrv\" (UniqueName: \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.304835 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.304750    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:34.586041 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:34.585990    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:26:34.588681 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:26:34.588652    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf55ca3_aed2_4fc3_9d4d_a97f456f6d31.slice/crio-346aea23624c2861601e192fe3c2e4773ec8853716e6d6933d47517d54712dad WatchSource:0}: Error finding container 346aea23624c2861601e192fe3c2e4773ec8853716e6d6933d47517d54712dad: Status 404 returned error can't find the container with id 346aea23624c2861601e192fe3c2e4773ec8853716e6d6933d47517d54712dad
Oct 30 19:26:35.219725 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:35.219679    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerStarted","Data":"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"}
Oct 30 19:26:35.220262 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:35.219744    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerStarted","Data":"346aea23624c2861601e192fe3c2e4773ec8853716e6d6933d47517d54712dad"}
Oct 30 19:26:36.226598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:36.226562    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerID="2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25" exitCode=0
Oct 30 19:26:36.227029 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:36.226650    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerDied","Data":"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"}
Oct 30 19:26:37.234796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:37.234757    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerStarted","Data":"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"}
Oct 30 19:26:37.235263 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:37.234821    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerStarted","Data":"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"}
Oct 30 19:26:37.235263 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:37.234849    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:37.260558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:37.260503    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podStartSLOduration=3.057516912 podStartE2EDuration="4.260486867s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.202969955s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:37.260054998 +0000 UTC m=+2983.838150690" watchObservedRunningTime="2026-10-30 19:26:37.260486867 +0000 UTC m=+2983.838582559"
Oct 30 19:26:38.239313 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:38.239277    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:38.240666 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:38.240640    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:26:39.243875 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:39.243834    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:26:39.962357 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:39.962306    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:44.248679 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:44.248645    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:26:44.249399 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:44.249360    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:26:49.963088 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:49.963040    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:50.537453 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:50.537403    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:26:53.806519 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:53.806402    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:26:53.813416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:53.813390    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:26:54.250285 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:26:54.250243    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:26:56.726362 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:26:56.726316    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:27:00.538911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:00.538877    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:27:04.250209 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:04.250115    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:06.797507 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:06.797472    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:27:06.797960 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:06.797729    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" containerID="cri-o://094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407" gracePeriod=30
Oct 30 19:27:06.797960 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:06.797801    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kube-rbac-proxy" containerID="cri-o://b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97" gracePeriod=30
Oct 30 19:27:07.372146 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:07.372112    4942 generic.go:356] "Generic (PLEG): container finished" podID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerID="b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97" exitCode=0
Oct 30 19:27:07.372323 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:07.372156    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerDied","Data":"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"}
Oct 30 19:27:09.958230 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:09.958187    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.73:8643/healthz\": dial tcp 10.132.0.73:8643: connect: connection refused"
Oct 30 19:27:10.537530 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:10.537487    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 19:27:11.865066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:11.865036    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:27:12.019138 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019100    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\") pod \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") "
Oct 30 19:27:12.019344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019160    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\") pod \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") "
Oct 30 19:27:12.019344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019179    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\") pod \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") "
Oct 30 19:27:12.019344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019208    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\") pod \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") "
Oct 30 19:27:12.019344 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019249    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\") pod \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\" (UID: \"30bb0f04-65a3-40ea-879c-0e8b3786b2e5\") "
Oct 30 19:27:12.019639 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019602    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert" pod "30bb0f04-65a3-40ea-879c-0e8b3786b2e5" (UID: "30bb0f04-65a3-40ea-879c-0e8b3786b2e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:12.019697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019604    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config" pod "30bb0f04-65a3-40ea-879c-0e8b3786b2e5" (UID: "30bb0f04-65a3-40ea-879c-0e8b3786b2e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:12.019697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.019647    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location" pod "30bb0f04-65a3-40ea-879c-0e8b3786b2e5" (UID: "30bb0f04-65a3-40ea-879c-0e8b3786b2e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:12.021548 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.021521    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls" pod "30bb0f04-65a3-40ea-879c-0e8b3786b2e5" (UID: "30bb0f04-65a3-40ea-879c-0e8b3786b2e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:12.021642 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.021562    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf" pod "30bb0f04-65a3-40ea-879c-0e8b3786b2e5" (UID: "30bb0f04-65a3-40ea-879c-0e8b3786b2e5"). InnerVolumeSpecName "kube-api-access-2gjsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:12.120124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.120081    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.120124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.120117    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kube-api-access-2gjsf\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.120124 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.120129    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.120370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.120139    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.120370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.120148    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30bb0f04-65a3-40ea-879c-0e8b3786b2e5-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.396664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.396574    4942 generic.go:356] "Generic (PLEG): container finished" podID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerID="094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407" exitCode=0
Oct 30 19:27:12.396664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.396618    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerDied","Data":"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"}
Oct 30 19:27:12.396664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.396641    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"
Oct 30 19:27:12.396913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.396668    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb" event={"ID":"30bb0f04-65a3-40ea-879c-0e8b3786b2e5","Type":"ContainerDied","Data":"bcb570426a23d1cd892ec8ddc4cd917e5663dd6fcc4a052496e9b7ebd5abcecc"}
Oct 30 19:27:12.396913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.396696    4942 scope.go:122] "RemoveContainer" containerID="b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"
Oct 30 19:27:12.406443 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.406418    4942 scope.go:122] "RemoveContainer" containerID="094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"
Oct 30 19:27:12.415697 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.415671    4942 scope.go:122] "RemoveContainer" containerID="8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"
Oct 30 19:27:12.421764 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.421732    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:27:12.425324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.425295    4942 scope.go:122] "RemoveContainer" containerID="b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"
Oct 30 19:27:12.425694 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:27:12.425668    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97\": container with ID starting with b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97 not found: ID does not exist" containerID="b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"
Oct 30 19:27:12.425768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.425703    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97"} err="failed to get container status \"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97\": rpc error: code = NotFound desc = could not find container \"b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97\": container with ID starting with b7a74558053325838cabcb1ee43f9d7f14f45090fbf7802ab2be9c18ba878c97 not found: ID does not exist"
Oct 30 19:27:12.425768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.425724    4942 scope.go:122] "RemoveContainer" containerID="094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"
Oct 30 19:27:12.425855 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.425837    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-rollback-predictor-777dfc6864-6d9hb"]
Oct 30 19:27:12.425984 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:27:12.425967    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407\": container with ID starting with 094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407 not found: ID does not exist" containerID="094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"
Oct 30 19:27:12.426055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.425995    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407"} err="failed to get container status \"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407\": rpc error: code = NotFound desc = could not find container \"094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407\": container with ID starting with 094a608786949ffd9d4faa054536bd3ce61e28ad13978d68b13815d0eadb3407 not found: ID does not exist"
Oct 30 19:27:12.426055 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.426039    4942 scope.go:122] "RemoveContainer" containerID="8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"
Oct 30 19:27:12.426292 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:27:12.426275    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed\": container with ID starting with 8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed not found: ID does not exist" containerID="8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"
Oct 30 19:27:12.426347 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:12.426294    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed"} err="failed to get container status \"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed\": rpc error: code = NotFound desc = could not find container \"8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed\": container with ID starting with 8364b238c24ce698dd442cc3a79cf517f7f16addf7963520c0fdc84cb04754ed not found: ID does not exist"
Oct 30 19:27:13.543857 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:13.543816    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" path="/var/lib/kubelet/pods/30bb0f04-65a3-40ea-879c-0e8b3786b2e5/volumes"
Oct 30 19:27:14.249251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.249213    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:14.423872 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.423837    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:27:14.424214 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424200    4942 cpu_manager.go:414] "containerMap: removing container" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kube-rbac-proxy"
Oct 30 19:27:14.424281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424215    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kube-rbac-proxy"
Oct 30 19:27:14.424281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424233    4942 cpu_manager.go:414] "containerMap: removing container" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="storage-initializer"
Oct 30 19:27:14.424281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424238    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="storage-initializer"
Oct 30 19:27:14.424281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424245    4942 cpu_manager.go:414] "containerMap: removing container" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container"
Oct 30 19:27:14.424281 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424250    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container"
Oct 30 19:27:14.424435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424312    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kserve-container"
Oct 30 19:27:14.424435 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.424325    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bb0f04-65a3-40ea-879c-0e8b3786b2e5" containerName="kube-rbac-proxy"
Oct 30 19:27:14.429388 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.429359    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.436900 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.436872    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.436993 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.436927    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.437076 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.437042    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.437143 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.437084    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvcd\" (UniqueName: \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.437192 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.437147    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.438174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.438150    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 30 19:27:14.439888 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.439861    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:27:14.439888 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.439878    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:27:14.440052 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.439954    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 19:27:14.440052 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.439991    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:27:14.440782 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.440761    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:27:14.440782 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.440775    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:27:14.452978 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.451907    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:27:14.538233 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538161    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538233 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538211    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538458 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538242    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538458 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538259    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvcd\" (UniqueName: \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538458 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538280    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538654 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538631    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.538940 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538917    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.539040 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.538978    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.540694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.540670    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.551641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.551615    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvcd\" (UniqueName: \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\") pod \"isvc-canary-stop-v2-predictor-755c557f5c-wvg8w\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.578686 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.578661    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:27:14.582524 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.582509    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.600850 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.600826    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:27:14.638594 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638566    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 19:27:14.638594 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638582    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.638752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638673    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.638752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638700    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqp7\" (UniqueName: \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.638752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638725    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.638752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.638740    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739430 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739403    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739436    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqp7\" (UniqueName: \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739464    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739482    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739570 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739537    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.739863 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.739841    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.740234 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.740216    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.740287 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.740219    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.741972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.741949    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.748581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.748557    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:14.758371 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.758334    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqp7\" (UniqueName: \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\") pod \"isvc-canary-stop-predictor-667bcbb7d8-jpn7j\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") " pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:14.899544 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:14.899498    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:15.036625 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.036592    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:27:15.040002 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:27:15.039976    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786b5752_9609_4788_8580_f6d7ec85813f.slice/crio-a40d1f8927a8896779c85cdbf7a5de584cdf5497d428030dd9ef2a79e0b06c19 WatchSource:0}: Error finding container a40d1f8927a8896779c85cdbf7a5de584cdf5497d428030dd9ef2a79e0b06c19: Status 404 returned error can't find the container with id a40d1f8927a8896779c85cdbf7a5de584cdf5497d428030dd9ef2a79e0b06c19
Oct 30 19:27:15.188830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.188796    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:27:15.191623 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:27:15.191595    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff0765d_03c0_49f3_a43a_5005d78432bf.slice/crio-b3b311a6695f6656051487beb9a8437942eb90163664eeb64aa1b35f74d091e1 WatchSource:0}: Error finding container b3b311a6695f6656051487beb9a8437942eb90163664eeb64aa1b35f74d091e1: Status 404 returned error can't find the container with id b3b311a6695f6656051487beb9a8437942eb90163664eeb64aa1b35f74d091e1
Oct 30 19:27:15.413926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.413824    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerStarted","Data":"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"}
Oct 30 19:27:15.413926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.413908    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerStarted","Data":"b3b311a6695f6656051487beb9a8437942eb90163664eeb64aa1b35f74d091e1"}
Oct 30 19:27:15.415753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.415723    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerStarted","Data":"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"}
Oct 30 19:27:15.415897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:15.415781    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerStarted","Data":"a40d1f8927a8896779c85cdbf7a5de584cdf5497d428030dd9ef2a79e0b06c19"}
Oct 30 19:27:19.436597 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:19.436565    4942 generic.go:356] "Generic (PLEG): container finished" podID="786b5752-9609-4788-8580-f6d7ec85813f" containerID="c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489" exitCode=0
Oct 30 19:27:19.436997 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:19.436640    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerDied","Data":"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"}
Oct 30 19:27:20.442305 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.442274    4942 generic.go:356] "Generic (PLEG): container finished" podID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerID="1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a" exitCode=0
Oct 30 19:27:20.442766 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.442349    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerDied","Data":"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"}
Oct 30 19:27:20.445581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.445563    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerStarted","Data":"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"}
Oct 30 19:27:20.445645 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.445614    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerStarted","Data":"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"}
Oct 30 19:27:20.445816 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.445799    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:20.493954 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:20.493904    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podStartSLOduration=2.597099508 podStartE2EDuration="6.493891705s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.896792197s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:20.493426704 +0000 UTC m=+3027.071522397" watchObservedRunningTime="2026-10-30 19:27:20.493891705 +0000 UTC m=+3027.071987396"
Oct 30 19:27:21.455977 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.455940    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerStarted","Data":"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"}
Oct 30 19:27:21.456406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.456291    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:21.456406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.456345    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerStarted","Data":"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"}
Oct 30 19:27:21.456579 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.456560    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:21.456725 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.456706    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:21.457495 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.457473    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:21.457797 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.457776    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:27:21.482432 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:21.482367    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podStartSLOduration=3.258270319 podStartE2EDuration="7.482349922s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.224079603s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:21.481238545 +0000 UTC m=+3028.059334270" watchObservedRunningTime="2026-10-30 19:27:21.482349922 +0000 UTC m=+3028.060445608"
Oct 30 19:27:22.460770 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:22.460725    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:27:22.461202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:22.460840    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:24.250123 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:24.250075    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:26.723704 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:27:26.723665    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:27:27.465054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:27.465021    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:27:27.465445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:27.465421    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:27:27.465558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:27.465447    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:27.465864 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:27.465841    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:27:34.249501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:34.249452    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:37.465553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:37.465501    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:37.465967 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:37.465933    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:27:44.249293 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:44.249242    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:45.536965 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:45.536926    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:47.465738 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:47.465698    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:47.466132 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:47.465849    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:27:55.537294 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:55.537250    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:27:56.723422 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:27:56.723375    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:27:57.465486 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:57.465445    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:27:57.465850 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:27:57.465824    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:28:05.541387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:05.541359    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:28:07.465620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:07.465582    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:28:07.466046 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:07.465794    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:28:13.986119 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:13.986085    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:28:13.986487 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:13.986403    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" containerID="cri-o://a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c" gracePeriod=30
Oct 30 19:28:13.986563 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:13.986492    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy" containerID="cri-o://4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51" gracePeriod=30
Oct 30 19:28:14.245021 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:14.244908    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 19:28:14.693335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:14.693297    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerID="4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51" exitCode=0
Oct 30 19:28:14.693492 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:14.693372    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerDied","Data":"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"}
Oct 30 19:28:15.537030 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:15.536968    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 19:28:17.466145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:17.466108    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:28:17.466145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:17.466126    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:28:19.244727 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.244685    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 19:28:19.329442 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.329416    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:28:19.366866 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.366825    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\") pod \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") "
Oct 30 19:28:19.367025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.366878    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\" (UniqueName: \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\") pod \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") "
Oct 30 19:28:19.367025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.366915    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\") pod \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") "
Oct 30 19:28:19.367025 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.366946    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\") pod \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") "
Oct 30 19:28:19.367213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.367047    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\") pod \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\" (UID: \"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31\") "
Oct 30 19:28:19.367213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.367160    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config" pod "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" (UID: "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:19.367316 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.367285    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:19.367414 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.367392    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert" pod "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" (UID: "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:19.367475 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.367426    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location" pod "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" (UID: "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:19.369063 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.369038    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls" pod "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" (UID: "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:19.369153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.369075    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv" pod "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" (UID: "3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31"). InnerVolumeSpecName "kube-api-access-5mdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:19.468510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.468475    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:19.468510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.468509    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdrv\" (UniqueName: \"kubernetes.io/projected/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-kube-api-access-5mdrv\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:19.468705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.468523    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:19.468705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.468537    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:19.717039 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.716979    4942 generic.go:356] "Generic (PLEG): container finished" podID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerID="a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c" exitCode=0
Oct 30 19:28:19.717200 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.717102    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"
Oct 30 19:28:19.717200 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.717109    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerDied","Data":"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"}
Oct 30 19:28:19.717200 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.717172    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k" event={"ID":"3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31","Type":"ContainerDied","Data":"346aea23624c2861601e192fe3c2e4773ec8853716e6d6933d47517d54712dad"}
Oct 30 19:28:19.717305 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.717202    4942 scope.go:122] "RemoveContainer" containerID="4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"
Oct 30 19:28:19.725694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.725623    4942 scope.go:122] "RemoveContainer" containerID="a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"
Oct 30 19:28:19.732853 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.732837    4942 scope.go:122] "RemoveContainer" containerID="2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"
Oct 30 19:28:19.737352 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.737332    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:28:19.740779 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.740761    4942 scope.go:122] "RemoveContainer" containerID="4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"
Oct 30 19:28:19.741125 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:19.741091    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51\": container with ID starting with 4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51 not found: ID does not exist" containerID="4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"
Oct 30 19:28:19.741197 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.741129    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51"} err="failed to get container status \"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51\": rpc error: code = NotFound desc = could not find container \"4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51\": container with ID starting with 4724c086f22c304aaaadb55d86f1dced24b391d876da0ddf6eba23613bfb8e51 not found: ID does not exist"
Oct 30 19:28:19.741197 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.741152    4942 scope.go:122] "RemoveContainer" containerID="a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"
Oct 30 19:28:19.741433 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:19.741413    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c\": container with ID starting with a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c not found: ID does not exist" containerID="a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"
Oct 30 19:28:19.741500 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.741442    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c"} err="failed to get container status \"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c\": rpc error: code = NotFound desc = could not find container \"a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c\": container with ID starting with a6c903ae09875c36be7dde51f83f88bf23e28eb91958f87abce0aafd0775403c not found: ID does not exist"
Oct 30 19:28:19.741500 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.741465    4942 scope.go:122] "RemoveContainer" containerID="2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"
Oct 30 19:28:19.741692 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:19.741676    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25\": container with ID starting with 2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25 not found: ID does not exist" containerID="2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"
Oct 30 19:28:19.741761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.741700    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25"} err="failed to get container status \"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25\": rpc error: code = NotFound desc = could not find container \"2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25\": container with ID starting with 2d2c6ae69f4ef559657074b08f1118af7703b9fd3c4aea784600cf1e9516bd25 not found: ID does not exist"
Oct 30 19:28:19.742090 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:19.742070    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-global-pass-predictor-7bd8dfb8df-pnm7k"]
Oct 30 19:28:21.543494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:21.543454    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" path="/var/lib/kubelet/pods/3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31/volumes"
Oct 30 19:28:26.724702 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:26.724661    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:28:27.465919 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:27.465879    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:28:27.466119 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:27.465885    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:28:28.764724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.764690    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765054    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="storage-initializer"
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765068    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="storage-initializer"
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765084    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy"
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765090    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy"
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765097    4942 cpu_manager.go:414] "containerMap: removing container" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container"
Oct 30 19:28:28.765104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765102    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container"
Oct 30 19:28:28.765292 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765175    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kube-rbac-proxy"
Oct 30 19:28:28.765292 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.765183    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf55ca3-aed2-4fc3-9d4d-a97f456f6d31" containerName="kserve-container"
Oct 30 19:28:28.768573 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.768555    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779661    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779713    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"storage-config\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779720    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779771    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779793    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779852    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:28:28.780035 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.779865    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:28:28.781551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.781527    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:28:28.843739 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.843696    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.843924 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.843754    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.843924 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.843842    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.844056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.843937    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.844056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.844003    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkzr\" (UniqueName: \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.944867 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.944839    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.944995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.944899    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.944995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.944932    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkzr\" (UniqueName: \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.945132 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.945003    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.945132 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.945053    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.945505 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.945485    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.945702 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.945675    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.945774 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.945724    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.947877 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.947850    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:28.958298 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:28.958276    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkzr\" (UniqueName: \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:29.087377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:29.087300    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:29.368213 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:29.368153    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:28:29.370681 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:28:29.370653    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f75f64_a235_49ca_9ece_06b7d4ff3273.slice/crio-cc935b06eff51dcd48319a67f084d3ca1ab4f880ac9a7c1329604e71ba771525 WatchSource:0}: Error finding container cc935b06eff51dcd48319a67f084d3ca1ab4f880ac9a7c1329604e71ba771525: Status 404 returned error can't find the container with id cc935b06eff51dcd48319a67f084d3ca1ab4f880ac9a7c1329604e71ba771525
Oct 30 19:28:29.766635 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:29.766597    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerStarted","Data":"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"}
Oct 30 19:28:29.767068 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:29.766655    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerStarted","Data":"cc935b06eff51dcd48319a67f084d3ca1ab4f880ac9a7c1329604e71ba771525"}
Oct 30 19:28:30.771954 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:30.771923    4942 generic.go:356] "Generic (PLEG): container finished" podID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerID="7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff" exitCode=0
Oct 30 19:28:30.772410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:30.772022    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerDied","Data":"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"}
Oct 30 19:28:31.778684 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:31.778647    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerStarted","Data":"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"}
Oct 30 19:28:31.779074 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:31.778710    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerStarted","Data":"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"}
Oct 30 19:28:31.779074 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:31.778745    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:31.804704 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:31.804652    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podStartSLOduration=2.6914586849999997 podStartE2EDuration="3.804636856s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.113178171s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:31.803957504 +0000 UTC m=+3098.382053198" watchObservedRunningTime="2026-10-30 19:28:31.804636856 +0000 UTC m=+3098.382732606"
Oct 30 19:28:32.783832 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:32.783799    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:32.785123 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:32.785094    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:28:33.787948 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:33.787907    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:28:37.465749 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:37.465711    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 19:28:37.466182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:37.465872    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 19:28:38.792399 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:38.792371    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:28:38.792861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:38.792837    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:28:42.537201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:42.537151    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:28:42.801424 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:42.801340    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:28:42.801620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:42.801594    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container" containerID="cri-o://949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24" gracePeriod=30
Oct 30 19:28:42.801705 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:42.801618    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kube-rbac-proxy" containerID="cri-o://7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef" gracePeriod=30
Oct 30 19:28:42.802479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:42.802153    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:28:43.836932 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:43.836897    4942 generic.go:356] "Generic (PLEG): container finished" podID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerID="7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef" exitCode=0
Oct 30 19:28:43.837330 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:43.836972    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerDied","Data":"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"}
Oct 30 19:28:44.530926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:44.530895    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:28:44.531180 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:44.531132    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container" containerID="cri-o://750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9" gracePeriod=30
Oct 30 19:28:44.531243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:44.531184    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kube-rbac-proxy" containerID="cri-o://2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd" gracePeriod=30
Oct 30 19:28:44.844204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:44.844112    4942 generic.go:356] "Generic (PLEG): container finished" podID="786b5752-9609-4788-8580-f6d7ec85813f" containerID="2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd" exitCode=0
Oct 30 19:28:44.844744 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:44.844198    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerDied","Data":"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"}
Oct 30 19:28:47.461330 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.461287    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.76:8643/healthz\": dial tcp 10.132.0.76:8643: connect: connection refused"
Oct 30 19:28:47.461710 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.461289    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.75:8643/healthz\": dial tcp 10.132.0.75:8643: connect: connection refused"
Oct 30 19:28:47.850595 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.850568    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:28:47.861405 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.861376    4942 generic.go:356] "Generic (PLEG): container finished" podID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerID="949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24" exitCode=0
Oct 30 19:28:47.861532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.861439    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"
Oct 30 19:28:47.861532 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.861513    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerDied","Data":"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"}
Oct 30 19:28:47.861609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.861574    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j" event={"ID":"fff0765d-03c0-49f3-a43a-5005d78432bf","Type":"ContainerDied","Data":"b3b311a6695f6656051487beb9a8437942eb90163664eeb64aa1b35f74d091e1"}
Oct 30 19:28:47.861641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.861612    4942 scope.go:122] "RemoveContainer" containerID="7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"
Oct 30 19:28:47.869839 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.869823    4942 scope.go:122] "RemoveContainer" containerID="949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"
Oct 30 19:28:47.877595 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.877555    4942 scope.go:122] "RemoveContainer" containerID="1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"
Oct 30 19:28:47.885805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.885784    4942 scope.go:122] "RemoveContainer" containerID="7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"
Oct 30 19:28:47.886084 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:47.886066    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef\": container with ID starting with 7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef not found: ID does not exist" containerID="7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"
Oct 30 19:28:47.886157 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.886094    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef"} err="failed to get container status \"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef\": rpc error: code = NotFound desc = could not find container \"7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef\": container with ID starting with 7792484b2b7768c5c996919726b9fa83b060ff100e38bccecd2c306b755898ef not found: ID does not exist"
Oct 30 19:28:47.886157 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.886110    4942 scope.go:122] "RemoveContainer" containerID="949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"
Oct 30 19:28:47.886346 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:47.886330    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24\": container with ID starting with 949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24 not found: ID does not exist" containerID="949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"
Oct 30 19:28:47.886386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.886351    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24"} err="failed to get container status \"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24\": rpc error: code = NotFound desc = could not find container \"949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24\": container with ID starting with 949874257d9f700a51fed1bdd419847f4d0a07b2c9b56df4e1f82b292e976c24 not found: ID does not exist"
Oct 30 19:28:47.886386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.886367    4942 scope.go:122] "RemoveContainer" containerID="1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"
Oct 30 19:28:47.886603 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:47.886587    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a\": container with ID starting with 1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a not found: ID does not exist" containerID="1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"
Oct 30 19:28:47.886644 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:47.886608    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a"} err="failed to get container status \"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a\": rpc error: code = NotFound desc = could not find container \"1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a\": container with ID starting with 1979a192ea12dc66ee51ed6a9e10f80901db9036214ee2c037980daafe9ff62a not found: ID does not exist"
Oct 30 19:28:48.009605 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009549    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\") pod \"fff0765d-03c0-49f3-a43a-5005d78432bf\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") "
Oct 30 19:28:48.009605 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009579    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\" (UniqueName: \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\") pod \"fff0765d-03c0-49f3-a43a-5005d78432bf\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") "
Oct 30 19:28:48.009605 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009603    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\") pod \"fff0765d-03c0-49f3-a43a-5005d78432bf\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") "
Oct 30 19:28:48.009805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009648    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\") pod \"fff0765d-03c0-49f3-a43a-5005d78432bf\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") "
Oct 30 19:28:48.009805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009673    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\") pod \"fff0765d-03c0-49f3-a43a-5005d78432bf\" (UID: \"fff0765d-03c0-49f3-a43a-5005d78432bf\") "
Oct 30 19:28:48.010018 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.009975    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert" pod "fff0765d-03c0-49f3-a43a-5005d78432bf" (UID: "fff0765d-03c0-49f3-a43a-5005d78432bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:48.010096 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.010042    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config" pod "fff0765d-03c0-49f3-a43a-5005d78432bf" (UID: "fff0765d-03c0-49f3-a43a-5005d78432bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:48.010139 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.010096    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location" pod "fff0765d-03c0-49f3-a43a-5005d78432bf" (UID: "fff0765d-03c0-49f3-a43a-5005d78432bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:48.011668 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.011641    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls" pod "fff0765d-03c0-49f3-a43a-5005d78432bf" (UID: "fff0765d-03c0-49f3-a43a-5005d78432bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:48.011803 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.011785    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7" pod "fff0765d-03c0-49f3-a43a-5005d78432bf" (UID: "fff0765d-03c0-49f3-a43a-5005d78432bf"). InnerVolumeSpecName "kube-api-access-mnqp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:48.110509 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.110476    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:48.110509 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.110506    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0765d-03c0-49f3-a43a-5005d78432bf-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:48.110698 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.110519    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0765d-03c0-49f3-a43a-5005d78432bf-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:48.110698 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.110529    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqp7\" (UniqueName: \"kubernetes.io/projected/fff0765d-03c0-49f3-a43a-5005d78432bf-kube-api-access-mnqp7\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:48.110698 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.110538    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff0765d-03c0-49f3-a43a-5005d78432bf-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:48.186303 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.186271    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:28:48.190945 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.190919    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-predictor-667bcbb7d8-jpn7j"]
Oct 30 19:28:48.793627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:48.793589    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:28:49.475977 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.475948    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:28:49.543051 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.542977    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" path="/var/lib/kubelet/pods/fff0765d-03c0-49f3-a43a-5005d78432bf/volumes"
Oct 30 19:28:49.624145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624116    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\") pod \"786b5752-9609-4788-8580-f6d7ec85813f\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") "
Oct 30 19:28:49.624297 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624148    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\") pod \"786b5752-9609-4788-8580-f6d7ec85813f\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") "
Oct 30 19:28:49.624297 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624225    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\" (UniqueName: \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\") pod \"786b5752-9609-4788-8580-f6d7ec85813f\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") "
Oct 30 19:28:49.624297 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624245    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\") pod \"786b5752-9609-4788-8580-f6d7ec85813f\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") "
Oct 30 19:28:49.624297 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624263    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\") pod \"786b5752-9609-4788-8580-f6d7ec85813f\" (UID: \"786b5752-9609-4788-8580-f6d7ec85813f\") "
Oct 30 19:28:49.624557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624526    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location" pod "786b5752-9609-4788-8580-f6d7ec85813f" (UID: "786b5752-9609-4788-8580-f6d7ec85813f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:49.624645 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624620    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert" pod "786b5752-9609-4788-8580-f6d7ec85813f" (UID: "786b5752-9609-4788-8580-f6d7ec85813f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:49.624700 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.624635    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config" pod "786b5752-9609-4788-8580-f6d7ec85813f" (UID: "786b5752-9609-4788-8580-f6d7ec85813f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:49.626384 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.626353    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls" pod "786b5752-9609-4788-8580-f6d7ec85813f" (UID: "786b5752-9609-4788-8580-f6d7ec85813f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:49.626501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.626434    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd" pod "786b5752-9609-4788-8580-f6d7ec85813f" (UID: "786b5752-9609-4788-8580-f6d7ec85813f"). InnerVolumeSpecName "kube-api-access-kmvcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:49.725563 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.725538    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvcd\" (UniqueName: \"kubernetes.io/projected/786b5752-9609-4788-8580-f6d7ec85813f-kube-api-access-kmvcd\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:49.725563 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.725560    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:49.725711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.725572    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/786b5752-9609-4788-8580-f6d7ec85813f-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:49.725711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.725581    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786b5752-9609-4788-8580-f6d7ec85813f-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:49.725711 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.725590    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/786b5752-9609-4788-8580-f6d7ec85813f-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:49.879956 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.879875    4942 generic.go:356] "Generic (PLEG): container finished" podID="786b5752-9609-4788-8580-f6d7ec85813f" containerID="750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9" exitCode=0
Oct 30 19:28:49.879956 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.879933    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"
Oct 30 19:28:49.880481 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.879932    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerDied","Data":"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"}
Oct 30 19:28:49.880481 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.880068    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w" event={"ID":"786b5752-9609-4788-8580-f6d7ec85813f","Type":"ContainerDied","Data":"a40d1f8927a8896779c85cdbf7a5de584cdf5497d428030dd9ef2a79e0b06c19"}
Oct 30 19:28:49.880481 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.880096    4942 scope.go:122] "RemoveContainer" containerID="2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"
Oct 30 19:28:49.889220 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.889200    4942 scope.go:122] "RemoveContainer" containerID="750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"
Oct 30 19:28:49.896549 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.896532    4942 scope.go:122] "RemoveContainer" containerID="c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"
Oct 30 19:28:49.903894 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.903869    4942 scope.go:122] "RemoveContainer" containerID="2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"
Oct 30 19:28:49.904202 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:49.904179    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd\": container with ID starting with 2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd not found: ID does not exist" containerID="2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"
Oct 30 19:28:49.904261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904215    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd"} err="failed to get container status \"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd\": rpc error: code = NotFound desc = could not find container \"2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd\": container with ID starting with 2673bb895df8596b2c22e7638f8ad995692a0f62df8da7b3b866a81fd90665fd not found: ID does not exist"
Oct 30 19:28:49.904261 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904237    4942 scope.go:122] "RemoveContainer" containerID="750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"
Oct 30 19:28:49.904387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904374    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:28:49.904505 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:49.904488    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9\": container with ID starting with 750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9 not found: ID does not exist" containerID="750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"
Oct 30 19:28:49.904551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904510    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9"} err="failed to get container status \"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9\": rpc error: code = NotFound desc = could not find container \"750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9\": container with ID starting with 750e5c6f375d866d6cbed9f6693d721c5ad8793f6346d0236c8ac46e1ea7bfd9 not found: ID does not exist"
Oct 30 19:28:49.904551 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904523    4942 scope.go:122] "RemoveContainer" containerID="c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"
Oct 30 19:28:49.904740 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:49.904726    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489\": container with ID starting with c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489 not found: ID does not exist" containerID="c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"
Oct 30 19:28:49.904782 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.904744    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489"} err="failed to get container status \"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489\": rpc error: code = NotFound desc = could not find container \"c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489\": container with ID starting with c4cc6f197ea7395fec08f86b75186bd61acfb2ecec30667e910e09e6ff315489 not found: ID does not exist"
Oct 30 19:28:49.908507 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:49.908488    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-canary-stop-v2-predictor-755c557f5c-wvg8w"]
Oct 30 19:28:51.378844 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.378810    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379175    4942 cpu_manager.go:414] "containerMap: removing container" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379188    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379198    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379203    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379217    4942 cpu_manager.go:414] "containerMap: removing container" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="storage-initializer"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379223    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="storage-initializer"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379229    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379234    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379249    4942 cpu_manager.go:414] "containerMap: removing container" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container"
Oct 30 19:28:51.379254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379253    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379268    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="storage-initializer"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379273    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="storage-initializer"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379324    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kserve-container"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379335    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="786b5752-9609-4788-8580-f6d7ec85813f" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379341    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kube-rbac-proxy"
Oct 30 19:28:51.379578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.379348    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff0765d-03c0-49f3-a43a-5005d78432bf" containerName="kserve-container"
Oct 30 19:28:51.384099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.384083    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.410260 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.410235    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:28:51.410926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.410903    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 30 19:28:51.411739 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.411716    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:28:51.412188 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.412162    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:28:51.412878 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.412859    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:28:51.412957 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.412926    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 19:28:51.413033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.412980    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:28:51.413033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.412982    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:28:51.541872 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.541840    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.541872 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.541876    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.542093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.542018    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.542093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.542045    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.542164 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.542119    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkhd\" (UniqueName: \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.542702 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.542682    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786b5752-9609-4788-8580-f6d7ec85813f" path="/var/lib/kubelet/pods/786b5752-9609-4788-8580-f6d7ec85813f/volumes"
Oct 30 19:28:51.642510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642417    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.642510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642447    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.642510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642482    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkhd\" (UniqueName: \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.642510 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642510    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.642861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642530    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.642961 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.642937    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.643277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.643247    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.643380 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.643247    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.645095 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.645077    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.658197 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.658169    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkhd\" (UniqueName: \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\") pod \"isvc-lightgbm-predictor-66999bf7f5-82j4v\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.700745 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.700724    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:28:51.987049 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:51.987003    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:28:51.989286 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:28:51.989259    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf141239e_b22a_404f_bf7a_b4ac6b9649bd.slice/crio-45b800176b12793f18e7da450267c123f3545a99f8a6ba71979d274d05aeda08 WatchSource:0}: Error finding container 45b800176b12793f18e7da450267c123f3545a99f8a6ba71979d274d05aeda08: Status 404 returned error can't find the container with id 45b800176b12793f18e7da450267c123f3545a99f8a6ba71979d274d05aeda08
Oct 30 19:28:52.895870 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:52.895780    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerStarted","Data":"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"}
Oct 30 19:28:52.895870 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:52.895841    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerStarted","Data":"45b800176b12793f18e7da450267c123f3545a99f8a6ba71979d274d05aeda08"}
Oct 30 19:28:54.064169 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:54.064135    4942 scope.go:122] "RemoveContainer" containerID="a19566a1056ccc3da94b722d819025a3da43154370fac55e7ec1253ca43ce0ac"
Oct 30 19:28:54.072461 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:54.072433    4942 scope.go:122] "RemoveContainer" containerID="e1ec5e708afecb1585754a9a06471f5fc789ccc5760c9947bed6cca534e84336"
Oct 30 19:28:54.080044 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:54.080025    4942 scope.go:122] "RemoveContainer" containerID="835c8dbcda6de3c8a472ef3d2114fd89276f420b7c0df8ca9d6c6ccbc383b519"
Oct 30 19:28:56.723657 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:28:56.723622    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:28:56.913819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:56.913788    4942 generic.go:356] "Generic (PLEG): container finished" podID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerID="2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9" exitCode=0
Oct 30 19:28:56.913991 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:56.913860    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerDied","Data":"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"}
Oct 30 19:28:56.915162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:56.915145    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:28:58.793365 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:28:58.793320    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:03.952279 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:03.952241    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerStarted","Data":"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"}
Oct 30 19:29:03.952777 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:03.952312    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerStarted","Data":"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"}
Oct 30 19:29:03.952909 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:03.952883    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:29:03.979837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:03.979784    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podStartSLOduration=2.180881054 podStartE2EDuration="12.979772055s" totalImagesPullingTime="6.87123774s" totalInitContainerRuntime="3.927653261s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:03.97945257 +0000 UTC m=+3130.557548263" watchObservedRunningTime="2026-10-30 19:29:03.979772055 +0000 UTC m=+3130.557867746"
Oct 30 19:29:04.957239 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:04.957207    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:29:04.958632 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:04.958604    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:05.961448 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:05.961407    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:08.792973 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:08.792929    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:10.966948 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:10.966914    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:29:10.967546 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:10.967521    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:18.793051 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:18.792980    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:20.967971 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:20.967926    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:26.726618 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:29:26.726576    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:29:28.792965 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:28.792917    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:30.968376 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:30.968327    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:38.793527 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:38.793479    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:40.968057 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:40.967976    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:48.793598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:48.793553    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 19:29:50.968056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:50.967987    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:29:56.727164 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:29:56.727121    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:29:58.793171 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:29:58.793138    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:30:00.968046 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:00.967983    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:08.847339 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:08.847300    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:30:08.847983 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:08.847579    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container" containerID="cri-o://95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61" gracePeriod=30
Oct 30 19:30:08.847983 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:08.847608    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kube-rbac-proxy" containerID="cri-o://7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6" gracePeriod=30
Oct 30 19:30:09.269090 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:09.269050    4942 generic.go:356] "Generic (PLEG): container finished" podID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerID="7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6" exitCode=0
Oct 30 19:30:09.269274 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:09.269110    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerDied","Data":"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"}
Oct 30 19:30:10.968387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:10.968330    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:13.788700 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:13.788653    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.77:8643/healthz\": dial tcp 10.132.0.77:8643: connect: connection refused"
Oct 30 19:30:14.619387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.619347    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:30:14.661712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.661659    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\" (UniqueName: \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\") pod \"46f75f64-a235-49ca-9ece-06b7d4ff3273\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") "
Oct 30 19:30:14.661901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.661801    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\") pod \"46f75f64-a235-49ca-9ece-06b7d4ff3273\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") "
Oct 30 19:30:14.661901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.661862    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\" (UniqueName: \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\") pod \"46f75f64-a235-49ca-9ece-06b7d4ff3273\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") "
Oct 30 19:30:14.661901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.661894    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\") pod \"46f75f64-a235-49ca-9ece-06b7d4ff3273\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") "
Oct 30 19:30:14.662093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.661939    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\") pod \"46f75f64-a235-49ca-9ece-06b7d4ff3273\" (UID: \"46f75f64-a235-49ca-9ece-06b7d4ff3273\") "
Oct 30 19:30:14.662367 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.662333    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert" pod "46f75f64-a235-49ca-9ece-06b7d4ff3273" (UID: "46f75f64-a235-49ca-9ece-06b7d4ff3273"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:14.662487 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.662342    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config" pod "46f75f64-a235-49ca-9ece-06b7d4ff3273" (UID: "46f75f64-a235-49ca-9ece-06b7d4ff3273"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:14.662487 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.662384    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location" pod "46f75f64-a235-49ca-9ece-06b7d4ff3273" (UID: "46f75f64-a235-49ca-9ece-06b7d4ff3273"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.664169 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.664138    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls" pod "46f75f64-a235-49ca-9ece-06b7d4ff3273" (UID: "46f75f64-a235-49ca-9ece-06b7d4ff3273"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:14.664300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.664136    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr" pod "46f75f64-a235-49ca-9ece-06b7d4ff3273" (UID: "46f75f64-a235-49ca-9ece-06b7d4ff3273"). InnerVolumeSpecName "kube-api-access-llkzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:14.763377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.763276    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.763377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.763312    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46f75f64-a235-49ca-9ece-06b7d4ff3273-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.763377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.763324    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.763377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.763334    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46f75f64-a235-49ca-9ece-06b7d4ff3273-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.763377 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:14.763344    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkzr\" (UniqueName: \"kubernetes.io/projected/46f75f64-a235-49ca-9ece-06b7d4ff3273-kube-api-access-llkzr\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:15.299264 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.299220    4942 generic.go:356] "Generic (PLEG): container finished" podID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerID="95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61" exitCode=0
Oct 30 19:30:15.299701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.299301    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"
Oct 30 19:30:15.299701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.299305    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerDied","Data":"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"}
Oct 30 19:30:15.299701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.299375    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj" event={"ID":"46f75f64-a235-49ca-9ece-06b7d4ff3273","Type":"ContainerDied","Data":"cc935b06eff51dcd48319a67f084d3ca1ab4f880ac9a7c1329604e71ba771525"}
Oct 30 19:30:15.299701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.299403    4942 scope.go:122] "RemoveContainer" containerID="7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"
Oct 30 19:30:15.309990 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.309961    4942 scope.go:122] "RemoveContainer" containerID="95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"
Oct 30 19:30:15.319301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.319275    4942 scope.go:122] "RemoveContainer" containerID="7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"
Oct 30 19:30:15.324170 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.324139    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:30:15.329410 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.329369    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-custom-pass-predictor-74d6dddd7c-rtcqj"]
Oct 30 19:30:15.329874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.329850    4942 scope.go:122] "RemoveContainer" containerID="7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"
Oct 30 19:30:15.330296 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:15.330271    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6\": container with ID starting with 7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6 not found: ID does not exist" containerID="7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"
Oct 30 19:30:15.330403 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.330306    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6"} err="failed to get container status \"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6\": rpc error: code = NotFound desc = could not find container \"7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6\": container with ID starting with 7ef8ab08fe2f7ae63206afe42515b47e1412a5475066f8b2152e1a938deb4ac6 not found: ID does not exist"
Oct 30 19:30:15.330403 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.330326    4942 scope.go:122] "RemoveContainer" containerID="95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"
Oct 30 19:30:15.330629 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:15.330613    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61\": container with ID starting with 95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61 not found: ID does not exist" containerID="95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"
Oct 30 19:30:15.330672 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.330633    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61"} err="failed to get container status \"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61\": rpc error: code = NotFound desc = could not find container \"95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61\": container with ID starting with 95b7ab9a995ffd1993fa530b8a714f64ce1ad9b129565fee12590c5dd351cb61 not found: ID does not exist"
Oct 30 19:30:15.330672 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.330648    4942 scope.go:122] "RemoveContainer" containerID="7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"
Oct 30 19:30:15.330889 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:15.330871    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff\": container with ID starting with 7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff not found: ID does not exist" containerID="7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"
Oct 30 19:30:15.330932 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.330894    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff"} err="failed to get container status \"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff\": rpc error: code = NotFound desc = could not find container \"7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff\": container with ID starting with 7020dccc8aac938e7b88b3468fd47d2f7b79c6e3dfa13e5d306daeba224ae8ff not found: ID does not exist"
Oct 30 19:30:15.542225 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:15.542185    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" path="/var/lib/kubelet/pods/46f75f64-a235-49ca-9ece-06b7d4ff3273/volumes"
Oct 30 19:30:20.968103 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:20.968063    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:23.753099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753061    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753395    4942 cpu_manager.go:414] "containerMap: removing container" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="storage-initializer"
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753405    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="storage-initializer"
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753419    4942 cpu_manager.go:414] "containerMap: removing container" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container"
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753424    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container"
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753442    4942 cpu_manager.go:414] "containerMap: removing container" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kube-rbac-proxy"
Oct 30 19:30:23.753479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.753447    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kube-rbac-proxy"
Oct 30 19:30:23.755059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.754249    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kserve-container"
Oct 30 19:30:23.755059 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.754281    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f75f64-a235-49ca-9ece-06b7d4ff3273" containerName="kube-rbac-proxy"
Oct 30 19:30:23.760755 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.760734    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.771678 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771643    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:30:23.771678 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771672    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:30:23.771841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771648    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:30:23.771841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771722    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 30 19:30:23.771841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771733    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"storage-config\""
Oct 30 19:30:23.771841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771749    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:30:23.771841 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.771805    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:30:23.772119 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.772099    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:30:23.836071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.836029    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.836221 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.836148    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzx4x\" (UniqueName: \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.836278 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.836212    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.836278 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.836239    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.836355 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.836321    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.936776 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.936733    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.936776 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.936775    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937042 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.936965    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.937068    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937181 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.937162    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzx4x\" (UniqueName: \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937257 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.937233    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937842 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.937822    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.937919 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.937827    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.939495 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.939471    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:23.950917 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:23.950885    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzx4x\" (UniqueName: \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") " pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:24.079804 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:24.079707    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:24.361498 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:24.361418    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:30:24.364972 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:30:24.364945    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4534ec4b_4aeb_4425_8af0_b262af1eade6.slice/crio-601fefec6e50e4c695384e851db84bea8fab38be5ebbf6c2f37a4e0df9f4af3e WatchSource:0}: Error finding container 601fefec6e50e4c695384e851db84bea8fab38be5ebbf6c2f37a4e0df9f4af3e: Status 404 returned error can't find the container with id 601fefec6e50e4c695384e851db84bea8fab38be5ebbf6c2f37a4e0df9f4af3e
Oct 30 19:30:25.344139 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:25.344099    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerStarted","Data":"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"}
Oct 30 19:30:25.344679 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:25.344162    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerStarted","Data":"601fefec6e50e4c695384e851db84bea8fab38be5ebbf6c2f37a4e0df9f4af3e"}
Oct 30 19:30:26.350128 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:26.350084    4942 generic.go:356] "Generic (PLEG): container finished" podID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerID="0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941" exitCode=0
Oct 30 19:30:26.350656 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:26.350171    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerDied","Data":"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"}
Oct 30 19:30:26.728140 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:26.728102    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:30:27.356823 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:27.356789    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerStarted","Data":"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"}
Oct 30 19:30:27.357294 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:27.356847    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerStarted","Data":"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"}
Oct 30 19:30:27.357294 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:27.356991    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:27.384205 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:27.384149    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podStartSLOduration=3.264577315 podStartE2EDuration="4.384137388s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.119560073s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.381982875 +0000 UTC m=+3213.960078566" watchObservedRunningTime="2026-10-30 19:30:27.384137388 +0000 UTC m=+3213.962233133"
Oct 30 19:30:28.361434 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:28.361399    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:28.362652 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:28.362627    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:30:29.365761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:29.365717    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:30:30.967742 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:30.967699    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:34.370443 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:34.370407    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:30:34.370899 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:34.370874    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:30:36.537335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:36.537295    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:44.371648 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:44.371605    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:30:46.538001 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:46.537964    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:30:52.501468 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:52.501435    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:30:52.501954 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:52.501684    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" containerID="cri-o://5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1" gracePeriod=30
Oct 30 19:30:52.501954 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:52.501718    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kube-rbac-proxy" containerID="cri-o://ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0" gracePeriod=30
Oct 30 19:30:53.472386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:53.472352    4942 generic.go:356] "Generic (PLEG): container finished" podID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerID="ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0" exitCode=0
Oct 30 19:30:53.472558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:53.472425    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerDied","Data":"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"}
Oct 30 19:30:54.371673 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:54.371579    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:30:55.962613 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:55.962572    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.78:8643/healthz\": dial tcp 10.132.0.78:8643: connect: connection refused"
Oct 30 19:30:56.537558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:56.537516    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 19:30:56.725564 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:56.725526    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:30:57.753105 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.753076    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:30:57.941707 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.941683    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\") pod \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") "
Oct 30 19:30:57.941885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.941738    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\") pod \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") "
Oct 30 19:30:57.941885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.941796    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\") pod \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") "
Oct 30 19:30:57.941885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.941825    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\") pod \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") "
Oct 30 19:30:57.941885 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.941862    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\" (UniqueName: \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\") pod \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\" (UID: \"f141239e-b22a-404f-bf7a-b4ac6b9649bd\") "
Oct 30 19:30:57.942157 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.942127    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location" pod "f141239e-b22a-404f-bf7a-b4ac6b9649bd" (UID: "f141239e-b22a-404f-bf7a-b4ac6b9649bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.942277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.942213    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.942277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.942230    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert" pod "f141239e-b22a-404f-bf7a-b4ac6b9649bd" (UID: "f141239e-b22a-404f-bf7a-b4ac6b9649bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.942277 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.942242    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config" pod "f141239e-b22a-404f-bf7a-b4ac6b9649bd" (UID: "f141239e-b22a-404f-bf7a-b4ac6b9649bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.944034 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.943995    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd" pod "f141239e-b22a-404f-bf7a-b4ac6b9649bd" (UID: "f141239e-b22a-404f-bf7a-b4ac6b9649bd"). InnerVolumeSpecName "kube-api-access-4pkhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.944034 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:57.943999    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls" pod "f141239e-b22a-404f-bf7a-b4ac6b9649bd" (UID: "f141239e-b22a-404f-bf7a-b4ac6b9649bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:58.042726 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.042695    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:58.042726 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.042722    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f141239e-b22a-404f-bf7a-b4ac6b9649bd-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:58.042911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.042734    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:58.042911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.042744    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkhd\" (UniqueName: \"kubernetes.io/projected/f141239e-b22a-404f-bf7a-b4ac6b9649bd-kube-api-access-4pkhd\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:58.494758 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.494719    4942 generic.go:356] "Generic (PLEG): container finished" podID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerID="5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1" exitCode=0
Oct 30 19:30:58.494963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.494779    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"
Oct 30 19:30:58.494963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.494799    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerDied","Data":"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"}
Oct 30 19:30:58.494963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.494863    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v" event={"ID":"f141239e-b22a-404f-bf7a-b4ac6b9649bd","Type":"ContainerDied","Data":"45b800176b12793f18e7da450267c123f3545a99f8a6ba71979d274d05aeda08"}
Oct 30 19:30:58.494963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.494900    4942 scope.go:122] "RemoveContainer" containerID="ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"
Oct 30 19:30:58.504101 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.504077    4942 scope.go:122] "RemoveContainer" containerID="5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"
Oct 30 19:30:58.512129 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.512110    4942 scope.go:122] "RemoveContainer" containerID="2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.520871    4942 scope.go:122] "RemoveContainer" containerID="ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:58.521254    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0\": container with ID starting with ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0 not found: ID does not exist" containerID="ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521289    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0"} err="failed to get container status \"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0\": rpc error: code = NotFound desc = could not find container \"ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0\": container with ID starting with ce462ccd8ddc64f57686f570c63f9a5610f3f1be4de0a8b4433f44ce14f142d0 not found: ID does not exist"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521311    4942 scope.go:122] "RemoveContainer" containerID="5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521484    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:58.521537    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1\": container with ID starting with 5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1 not found: ID does not exist" containerID="5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521558    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1"} err="failed to get container status \"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1\": rpc error: code = NotFound desc = could not find container \"5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1\": container with ID starting with 5d2c57c6c697b2834f9e99679a08092484d8fe3a5341f10f2beac4d27af303e1 not found: ID does not exist"
Oct 30 19:30:58.521689 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521576    4942 scope.go:122] "RemoveContainer" containerID="2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"
Oct 30 19:30:58.522000 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:30:58.521831    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9\": container with ID starting with 2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9 not found: ID does not exist" containerID="2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"
Oct 30 19:30:58.522000 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.521856    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9"} err="failed to get container status \"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9\": rpc error: code = NotFound desc = could not find container \"2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9\": container with ID starting with 2c45e6bef6306aa91f08ba5c0eef0d6aecb225d1d6c314211d1b718e5f584ef9 not found: ID does not exist"
Oct 30 19:30:58.524291 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:58.524267    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-predictor-66999bf7f5-82j4v"]
Oct 30 19:30:59.541845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:30:59.541808    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" path="/var/lib/kubelet/pods/f141239e-b22a-404f-bf7a-b4ac6b9649bd/volumes"
Oct 30 19:31:00.028964 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.028933    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:31:00.029284 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029272    4942 cpu_manager.go:414] "containerMap: removing container" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="storage-initializer"
Oct 30 19:31:00.029334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029286    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="storage-initializer"
Oct 30 19:31:00.029334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029296    4942 cpu_manager.go:414] "containerMap: removing container" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container"
Oct 30 19:31:00.029334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029302    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container"
Oct 30 19:31:00.029334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029317    4942 cpu_manager.go:414] "containerMap: removing container" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kube-rbac-proxy"
Oct 30 19:31:00.029334 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029323    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kube-rbac-proxy"
Oct 30 19:31:00.029489 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029400    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kserve-container"
Oct 30 19:31:00.029489 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.029409    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="f141239e-b22a-404f-bf7a-b4ac6b9649bd" containerName="kube-rbac-proxy"
Oct 30 19:31:00.034209 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.034185    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.039138 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.039117    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:31:00.042090 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.042044    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:31:00.044394 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.044369    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:00.044502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.044488    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:31:00.044578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.044564    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Oct 30 19:31:00.045771 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.045752    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:31:00.139712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.139678    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:00.139830 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.139730    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:31:00.159718 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.159677    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.159838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.159731    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.159838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.159791    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb6k\" (UniqueName: \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.159916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.159845    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.159958 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.159945    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.260927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.260893    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261111 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.260943    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261111 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.260967    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261111 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.260982    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb6k\" (UniqueName: \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261111 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.261000    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261538 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.261513    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261835 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.261810    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.261874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.261810    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.263387 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.263367    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.273786 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.273762    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb6k\" (UniqueName: \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\") pod \"isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.354440 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.354359    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:00.632437 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:00.632392    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:31:00.635363 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:31:00.635334    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caa3049_8628_43b7_8326_8bb4ae5a8a2d.slice/crio-c327ce828792d81788025a93ee03980252fdf333007e2370107d58f19072a4ff WatchSource:0}: Error finding container c327ce828792d81788025a93ee03980252fdf333007e2370107d58f19072a4ff: Status 404 returned error can't find the container with id c327ce828792d81788025a93ee03980252fdf333007e2370107d58f19072a4ff
Oct 30 19:31:01.515020 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:01.514982    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerStarted","Data":"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"}
Oct 30 19:31:01.515223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:01.515060    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerStarted","Data":"c327ce828792d81788025a93ee03980252fdf333007e2370107d58f19072a4ff"}
Oct 30 19:31:04.371458 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:04.371419    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:31:05.534961 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:05.534926    4942 generic.go:356] "Generic (PLEG): container finished" podID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerID="2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de" exitCode=0
Oct 30 19:31:05.535389 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:05.534992    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerDied","Data":"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"}
Oct 30 19:31:06.541968 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:06.541928    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerStarted","Data":"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"}
Oct 30 19:31:06.542503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:06.542022    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerStarted","Data":"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"}
Oct 30 19:31:06.542503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:06.542248    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:06.569737 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:06.569685    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podStartSLOduration=2.527243352 podStartE2EDuration="6.569671736s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.042428384s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:06.56723602 +0000 UTC m=+3253.145331712" watchObservedRunningTime="2026-10-30 19:31:06.569671736 +0000 UTC m=+3253.147767496"
Oct 30 19:31:07.546875 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:07.546843    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:07.548614 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:07.548587    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:08.551938 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:08.551898    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:13.556963 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:13.556931    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:31:13.557466 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:13.557441    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:14.371641 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:14.371603    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:31:23.557860 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:23.557815    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:24.372056 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:24.371972    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:31:26.728210 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:31:26.728168    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:31:33.557901 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:33.557862    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:34.371312 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:34.371270    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:31:40.537328 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:40.537286    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 19:31:43.557657 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:43.557618    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:50.538223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:50.538188    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:31:53.558031 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.557964    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:31:53.833267 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.833118    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:31:53.841536 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.841498    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:31:53.921820 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.921780    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:31:53.922226 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.922102    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container" containerID="cri-o://3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee" gracePeriod=30
Oct 30 19:31:53.922226 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:53.922111    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kube-rbac-proxy" containerID="cri-o://482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181" gracePeriod=30
Oct 30 19:31:54.365985 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:54.365936    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.79:8643/healthz\": dial tcp 10.132.0.79:8643: connect: connection refused"
Oct 30 19:31:54.764428 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:54.764392    4942 generic.go:356] "Generic (PLEG): container finished" podID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerID="482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181" exitCode=0
Oct 30 19:31:54.764856 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:54.764474    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerDied","Data":"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"}
Oct 30 19:31:56.724476 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:31:56.724432    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:31:58.769071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.769044    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:31:58.782333 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.782312    4942 generic.go:356] "Generic (PLEG): container finished" podID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerID="3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee" exitCode=0
Oct 30 19:31:58.782441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.782341    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerDied","Data":"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"}
Oct 30 19:31:58.782441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.782373    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"
Oct 30 19:31:58.782441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.782382    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2" event={"ID":"4534ec4b-4aeb-4425-8af0-b262af1eade6","Type":"ContainerDied","Data":"601fefec6e50e4c695384e851db84bea8fab38be5ebbf6c2f37a4e0df9f4af3e"}
Oct 30 19:31:58.782441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.782408    4942 scope.go:122] "RemoveContainer" containerID="482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"
Oct 30 19:31:58.790116 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.790099    4942 scope.go:122] "RemoveContainer" containerID="3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"
Oct 30 19:31:58.797214 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.797187    4942 scope.go:122] "RemoveContainer" containerID="0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"
Oct 30 19:31:58.805382 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.805351    4942 scope.go:122] "RemoveContainer" containerID="482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"
Oct 30 19:31:58.805662 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:31:58.805643    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181\": container with ID starting with 482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181 not found: ID does not exist" containerID="482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"
Oct 30 19:31:58.805728 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.805670    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181"} err="failed to get container status \"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181\": rpc error: code = NotFound desc = could not find container \"482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181\": container with ID starting with 482ebd638faea62e8752efb65203ceff329061406b3abcf0b77ebe8de3f0a181 not found: ID does not exist"
Oct 30 19:31:58.805728 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.805686    4942 scope.go:122] "RemoveContainer" containerID="3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"
Oct 30 19:31:58.805924 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:31:58.805910    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee\": container with ID starting with 3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee not found: ID does not exist" containerID="3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"
Oct 30 19:31:58.805975 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.805928    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee"} err="failed to get container status \"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee\": rpc error: code = NotFound desc = could not find container \"3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee\": container with ID starting with 3cd860b62fcf769c6aac8d7399b85217ece609b75e5e9687f205fb15ba5ca1ee not found: ID does not exist"
Oct 30 19:31:58.805975 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.805941    4942 scope.go:122] "RemoveContainer" containerID="0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"
Oct 30 19:31:58.806218 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:31:58.806201    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941\": container with ID starting with 0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941 not found: ID does not exist" containerID="0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"
Oct 30 19:31:58.806269 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.806222    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941"} err="failed to get container status \"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941\": rpc error: code = NotFound desc = could not find container \"0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941\": container with ID starting with 0989073a46df0c5afe7a352f1db35e842d5bbf93765bce7f20bcabea62b63941 not found: ID does not exist"
Oct 30 19:31:58.834655 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.834633    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\") pod \"4534ec4b-4aeb-4425-8af0-b262af1eade6\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") "
Oct 30 19:31:58.834761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.834678    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\" (UniqueName: \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\") pod \"4534ec4b-4aeb-4425-8af0-b262af1eade6\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") "
Oct 30 19:31:58.834761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.834703    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\") pod \"4534ec4b-4aeb-4425-8af0-b262af1eade6\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") "
Oct 30 19:31:58.834761 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.834751    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\") pod \"4534ec4b-4aeb-4425-8af0-b262af1eade6\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") "
Oct 30 19:31:58.834931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.834770    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\") pod \"4534ec4b-4aeb-4425-8af0-b262af1eade6\" (UID: \"4534ec4b-4aeb-4425-8af0-b262af1eade6\") "
Oct 30 19:31:58.835104 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.835071    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location" pod "4534ec4b-4aeb-4425-8af0-b262af1eade6" (UID: "4534ec4b-4aeb-4425-8af0-b262af1eade6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:58.835207 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.835119    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert" pod "4534ec4b-4aeb-4425-8af0-b262af1eade6" (UID: "4534ec4b-4aeb-4425-8af0-b262af1eade6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:58.835207 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.835132    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config" pod "4534ec4b-4aeb-4425-8af0-b262af1eade6" (UID: "4534ec4b-4aeb-4425-8af0-b262af1eade6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:58.836681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.836658    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x" pod "4534ec4b-4aeb-4425-8af0-b262af1eade6" (UID: "4534ec4b-4aeb-4425-8af0-b262af1eade6"). InnerVolumeSpecName "kube-api-access-gzx4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:58.836762 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.836739    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls" pod "4534ec4b-4aeb-4425-8af0-b262af1eade6" (UID: "4534ec4b-4aeb-4425-8af0-b262af1eade6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:58.935995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.935962    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:58.935995 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.935992    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4534ec4b-4aeb-4425-8af0-b262af1eade6-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:58.936201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.936002    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzx4x\" (UniqueName: \"kubernetes.io/projected/4534ec4b-4aeb-4425-8af0-b262af1eade6-kube-api-access-gzx4x\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:58.936201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.936037    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4534ec4b-4aeb-4425-8af0-b262af1eade6-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:58.936201 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:58.936047    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4534ec4b-4aeb-4425-8af0-b262af1eade6-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:59.110322 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:59.110289    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:31:59.112678 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:59.112653    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-sklearn-s3-tls-serving-pass-predictor-598978cf6d-rdnp2"]
Oct 30 19:31:59.541796 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:31:59.541764    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" path="/var/lib/kubelet/pods/4534ec4b-4aeb-4425-8af0-b262af1eade6/volumes"
Oct 30 19:32:03.558270 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:03.558232    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:32:10.687335 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687297    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687637    4942 cpu_manager.go:414] "containerMap: removing container" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="storage-initializer"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687648    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="storage-initializer"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687659    4942 cpu_manager.go:414] "containerMap: removing container" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687664    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687671    4942 cpu_manager.go:414] "containerMap: removing container" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kube-rbac-proxy"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687676    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kube-rbac-proxy"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687739    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kserve-container"
Oct 30 19:32:10.687754 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.687746    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="4534ec4b-4aeb-4425-8af0-b262af1eade6" containerName="kube-rbac-proxy"
Oct 30 19:32:10.692627 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.692606    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.703197    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.703253    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.703823    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.703935    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.704364    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.704437    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.704478    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-predictor-serving-cert\""
Oct 30 19:32:10.705918 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.705075    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:32:10.834599 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.834570    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.834744 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.834681    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.834744 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.834710    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77dq\" (UniqueName: \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.834829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.834777    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.834829 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.834798    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.935634 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.935597    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77dq\" (UniqueName: \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.935819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.935642    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.935819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.935669    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.935819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.935697    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.935819 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.935805    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.936112 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.936090    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.936406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.936388    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.936492 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.936405    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.938310 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.938257    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:10.949268 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:10.949248    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77dq\" (UniqueName: \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\") pod \"isvc-paddle-predictor-85767878c8-hrtpd\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:11.012469 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:11.012440    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:11.297408 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:11.297329    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:32:11.300668 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:32:11.300637    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6d72a2_37af_4460_8990_696343fef0ae.slice/crio-6c726fb4730de7e00c29cd14c733b56b59f49e7d00e3ca4329c0aa1abed10fc4 WatchSource:0}: Error finding container 6c726fb4730de7e00c29cd14c733b56b59f49e7d00e3ca4329c0aa1abed10fc4: Status 404 returned error can't find the container with id 6c726fb4730de7e00c29cd14c733b56b59f49e7d00e3ca4329c0aa1abed10fc4
Oct 30 19:32:11.846066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:11.846026    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerStarted","Data":"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"}
Oct 30 19:32:11.846566 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:11.846083    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerStarted","Data":"6c726fb4730de7e00c29cd14c733b56b59f49e7d00e3ca4329c0aa1abed10fc4"}
Oct 30 19:32:13.558369 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:13.558334    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:32:16.868636 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:16.868609    4942 generic.go:356] "Generic (PLEG): container finished" podID="fb6d72a2-37af-4460-8990-696343fef0ae" containerID="3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef" exitCode=0
Oct 30 19:32:16.869029 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:16.868690    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerDied","Data":"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"}
Oct 30 19:32:23.558177 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:23.558130    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:32:26.729330 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:32:26.729283    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:32:28.932520 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:28.932460    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerStarted","Data":"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"}
Oct 30 19:32:28.932926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:28.932559    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerStarted","Data":"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"}
Oct 30 19:32:28.932926 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:28.932763    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:28.957495 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:28.957436    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podStartSLOduration=1.883993969 podStartE2EDuration="18.957421461s" totalImagesPullingTime="11.620430877s" totalInitContainerRuntime="5.452996615s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:28.95694739 +0000 UTC m=+3335.535043098" watchObservedRunningTime="2026-10-30 19:32:28.957421461 +0000 UTC m=+3335.535517155"
Oct 30 19:32:29.936792 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:29.936758    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:29.938221 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:29.938190    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:32:30.941324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:30.941282    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:32:33.557971 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:33.557924    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 19:32:35.946341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:35.946311    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:32:35.946875 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:35.946844    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:32:40.537695 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:40.537666    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:32:45.946795 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:45.946750    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:32:52.319079 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:52.319041    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:32:52.319571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:52.319281    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container" containerID="cri-o://21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988" gracePeriod=30
Oct 30 19:32:52.319650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:52.319590    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kube-rbac-proxy" containerID="cri-o://8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a" gracePeriod=30
Oct 30 19:32:53.040075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:53.040035    4942 generic.go:356] "Generic (PLEG): container finished" podID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerID="8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a" exitCode=0
Oct 30 19:32:53.040246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:53.040095    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerDied","Data":"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"}
Oct 30 19:32:53.552539 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:53.552496    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.80:8643/healthz\": dial tcp 10.132.0.80:8643: connect: connection refused"
Oct 30 19:32:55.947198 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:55.947149    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:32:56.725707 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:32:56.725669    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:32:57.670585 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.670550    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:32:57.831273 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831202    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\" (UniqueName: \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\") pod \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") "
Oct 30 19:32:57.831273 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831272    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\") pod \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") "
Oct 30 19:32:57.831556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831294    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\") pod \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") "
Oct 30 19:32:57.831556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831333    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\") pod \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") "
Oct 30 19:32:57.831556 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831377    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\") pod \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\" (UID: \"6caa3049-8628-43b7-8326-8bb4ae5a8a2d\") "
Oct 30 19:32:57.831720 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831649    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location" pod "6caa3049-8628-43b7-8326-8bb4ae5a8a2d" (UID: "6caa3049-8628-43b7-8326-8bb4ae5a8a2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:57.831772 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831741    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert" pod "6caa3049-8628-43b7-8326-8bb4ae5a8a2d" (UID: "6caa3049-8628-43b7-8326-8bb4ae5a8a2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:57.831826 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.831805    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config" pod "6caa3049-8628-43b7-8326-8bb4ae5a8a2d" (UID: "6caa3049-8628-43b7-8326-8bb4ae5a8a2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:57.833386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.833359    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls" pod "6caa3049-8628-43b7-8326-8bb4ae5a8a2d" (UID: "6caa3049-8628-43b7-8326-8bb4ae5a8a2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:57.833495 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.833441    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k" pod "6caa3049-8628-43b7-8326-8bb4ae5a8a2d" (UID: "6caa3049-8628-43b7-8326-8bb4ae5a8a2d"). InnerVolumeSpecName "kube-api-access-dwb6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:57.932687 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.932659    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:57.932687 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.932683    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:57.932843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.932697    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:57.932843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.932710    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:57.932843 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:57.932722    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwb6k\" (UniqueName: \"kubernetes.io/projected/6caa3049-8628-43b7-8326-8bb4ae5a8a2d-kube-api-access-dwb6k\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:58.065359 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.065317    4942 generic.go:356] "Generic (PLEG): container finished" podID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerID="21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988" exitCode=0
Oct 30 19:32:58.065545 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.065386    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"
Oct 30 19:32:58.065545 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.065401    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerDied","Data":"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"}
Oct 30 19:32:58.065545 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.065465    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf" event={"ID":"6caa3049-8628-43b7-8326-8bb4ae5a8a2d","Type":"ContainerDied","Data":"c327ce828792d81788025a93ee03980252fdf333007e2370107d58f19072a4ff"}
Oct 30 19:32:58.065545 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.065493    4942 scope.go:122] "RemoveContainer" containerID="8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"
Oct 30 19:32:58.074506 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.074490    4942 scope.go:122] "RemoveContainer" containerID="21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"
Oct 30 19:32:58.082103 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.082088    4942 scope.go:122] "RemoveContainer" containerID="2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"
Oct 30 19:32:58.089653 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.089557    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:32:58.089708 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.089679    4942 scope.go:122] "RemoveContainer" containerID="8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"
Oct 30 19:32:58.089974 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:32:58.089952    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a\": container with ID starting with 8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a not found: ID does not exist" containerID="8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"
Oct 30 19:32:58.090120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.089983    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a"} err="failed to get container status \"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a\": rpc error: code = NotFound desc = could not find container \"8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a\": container with ID starting with 8dd865a789a2e2c7ba933ce558f853cc5ae949e3b812ec497572f938fe04fb0a not found: ID does not exist"
Oct 30 19:32:58.090120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.089998    4942 scope.go:122] "RemoveContainer" containerID="21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"
Oct 30 19:32:58.090918 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:32:58.090898    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988\": container with ID starting with 21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988 not found: ID does not exist" containerID="21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"
Oct 30 19:32:58.090972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.090927    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988"} err="failed to get container status \"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988\": rpc error: code = NotFound desc = could not find container \"21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988\": container with ID starting with 21d86d214f51209613741c2b7c9fb77839533a33c4547e680afc248cea1ec988 not found: ID does not exist"
Oct 30 19:32:58.090972 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.090942    4942 scope.go:122] "RemoveContainer" containerID="2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"
Oct 30 19:32:58.091241 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:32:58.091216    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de\": container with ID starting with 2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de not found: ID does not exist" containerID="2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"
Oct 30 19:32:58.091287 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.091247    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de"} err="failed to get container status \"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de\": rpc error: code = NotFound desc = could not find container \"2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de\": container with ID starting with 2c5f7850a323528be8fa63f7de0cca27a6ff2dbdbe616ed6361d4d0e086b70de not found: ID does not exist"
Oct 30 19:32:58.094231 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:58.094210    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-runtime-predictor-854cf4d7bc-7mmlf"]
Oct 30 19:32:59.542112 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.542080    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" path="/var/lib/kubelet/pods/6caa3049-8628-43b7-8326-8bb4ae5a8a2d/volumes"
Oct 30 19:32:59.850394 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850312    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:32:59.850808 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850788    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container"
Oct 30 19:32:59.850893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850808    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container"
Oct 30 19:32:59.850893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850820    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="storage-initializer"
Oct 30 19:32:59.850893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850828    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="storage-initializer"
Oct 30 19:32:59.850893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850835    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kube-rbac-proxy"
Oct 30 19:32:59.850893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850840    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kube-rbac-proxy"
Oct 30 19:32:59.851070 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850907    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kserve-container"
Oct 30 19:32:59.851070 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.850916    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caa3049-8628-43b7-8326-8bb4ae5a8a2d" containerName="kube-rbac-proxy"
Oct 30 19:32:59.989835 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.989800    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:32:59.990028 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.989929    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:32:59.994701 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.994673    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:32:59.994856 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.994779    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:32:59.996080 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.996053    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:32:59.996216 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.996185    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:32:59.996286 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.996111    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:32:59.996286 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.996197    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 30 19:32:59.996286 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:32:59.996268    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:33:00.152285 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.152208    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h56j\" (UniqueName: \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.152433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.152331    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.152433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.152368    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.152433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.152387    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.152433 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.152419    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253262 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253225    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h56j\" (UniqueName: \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253279    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253303    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253320    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253341    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.253772 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.253750    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.254163 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.254137    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.254237 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.254137    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.255941 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.255893    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.266481 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.266457    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h56j\" (UniqueName: \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\") pod \"isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") " pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.365217 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.365186    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:00.665499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:00.665464    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:33:00.669224 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:33:00.669191    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da72633_02d6_47a7_a8bb_10c6827bfa3b.slice/crio-8ba310a76e77b221ffb3f35fb6e16ce283c97447ea3c314a86e8f494ebda72b0 WatchSource:0}: Error finding container 8ba310a76e77b221ffb3f35fb6e16ce283c97447ea3c314a86e8f494ebda72b0: Status 404 returned error can't find the container with id 8ba310a76e77b221ffb3f35fb6e16ce283c97447ea3c314a86e8f494ebda72b0
Oct 30 19:33:01.085581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:01.085492    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerStarted","Data":"0dda07aa70e245503df4ee1d1ab8c75710900095b1b4965d8f1b0fcdae50fac2"}
Oct 30 19:33:01.085581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:01.085573    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerStarted","Data":"8ba310a76e77b221ffb3f35fb6e16ce283c97447ea3c314a86e8f494ebda72b0"}
Oct 30 19:33:05.104486 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:05.104447    4942 generic.go:356] "Generic (PLEG): container finished" podID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerID="0dda07aa70e245503df4ee1d1ab8c75710900095b1b4965d8f1b0fcdae50fac2" exitCode=0
Oct 30 19:33:05.104855 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:05.104517    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerDied","Data":"0dda07aa70e245503df4ee1d1ab8c75710900095b1b4965d8f1b0fcdae50fac2"}
Oct 30 19:33:05.946935 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:05.946894    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 19:33:06.111273 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:06.111226    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerStarted","Data":"39fb7bb79b099161c8458a541b448163a631533d03f1fcc0a5179ff4bf3cb8a9"}
Oct 30 19:33:06.111651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:06.111310    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerStarted","Data":"fa324df5718ecb609fa870c088fea82efdacba064f659ee0eb7bcbab27d771c6"}
Oct 30 19:33:06.111651 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:06.111624    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:06.139255 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:06.139204    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" podStartSLOduration=3.004943829 podStartE2EDuration="7.139190835s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.134247006s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:06.137198089 +0000 UTC m=+3372.715293818" watchObservedRunningTime="2026-10-30 19:33:06.139190835 +0000 UTC m=+3372.717286526"
Oct 30 19:33:07.115766 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:07.115734    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:13.125439 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:13.125412    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:15.947240 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:15.947205    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:33:22.230693 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:22.230656    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:33:22.231193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:22.230953    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container" containerID="cri-o://5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9" gracePeriod=30
Oct 30 19:33:22.231193 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:22.231050    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kube-rbac-proxy" containerID="cri-o://b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec" gracePeriod=30
Oct 30 19:33:23.184237 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:23.184200    4942 generic.go:356] "Generic (PLEG): container finished" podID="fb6d72a2-37af-4460-8990-696343fef0ae" containerID="b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec" exitCode=0
Oct 30 19:33:23.184424 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:23.184260    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerDied","Data":"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"}
Oct 30 19:33:24.874192 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.874163    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:33:24.956756 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.956729    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\") pod \"fb6d72a2-37af-4460-8990-696343fef0ae\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") "
Oct 30 19:33:24.956911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.956786    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\") pod \"fb6d72a2-37af-4460-8990-696343fef0ae\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") "
Oct 30 19:33:24.956911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.956819    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\") pod \"fb6d72a2-37af-4460-8990-696343fef0ae\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") "
Oct 30 19:33:24.956911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.956845    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\" (UniqueName: \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\") pod \"fb6d72a2-37af-4460-8990-696343fef0ae\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") "
Oct 30 19:33:24.956911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.956885    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\") pod \"fb6d72a2-37af-4460-8990-696343fef0ae\" (UID: \"fb6d72a2-37af-4460-8990-696343fef0ae\") "
Oct 30 19:33:24.957187 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.957164    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config" pod "fb6d72a2-37af-4460-8990-696343fef0ae" (UID: "fb6d72a2-37af-4460-8990-696343fef0ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:24.957251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.957225    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert" pod "fb6d72a2-37af-4460-8990-696343fef0ae" (UID: "fb6d72a2-37af-4460-8990-696343fef0ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:24.958877 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.958850    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq" pod "fb6d72a2-37af-4460-8990-696343fef0ae" (UID: "fb6d72a2-37af-4460-8990-696343fef0ae"). InnerVolumeSpecName "kube-api-access-w77dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:24.958977 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.958908    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls" pod "fb6d72a2-37af-4460-8990-696343fef0ae" (UID: "fb6d72a2-37af-4460-8990-696343fef0ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:24.966547 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:24.966524    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location" pod "fb6d72a2-37af-4460-8990-696343fef0ae" (UID: "fb6d72a2-37af-4460-8990-696343fef0ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:25.057708 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.057678    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.057708 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.057704    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6d72a2-37af-4460-8990-696343fef0ae-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.057874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.057717    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb6d72a2-37af-4460-8990-696343fef0ae-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.057874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.057727    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77dq\" (UniqueName: \"kubernetes.io/projected/fb6d72a2-37af-4460-8990-696343fef0ae-kube-api-access-w77dq\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.057874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.057736    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb6d72a2-37af-4460-8990-696343fef0ae-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.194892 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.194851    4942 generic.go:356] "Generic (PLEG): container finished" podID="fb6d72a2-37af-4460-8990-696343fef0ae" containerID="5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9" exitCode=0
Oct 30 19:33:25.195071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.194921    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"
Oct 30 19:33:25.195071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.194929    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerDied","Data":"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"}
Oct 30 19:33:25.195071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.194994    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd" event={"ID":"fb6d72a2-37af-4460-8990-696343fef0ae","Type":"ContainerDied","Data":"6c726fb4730de7e00c29cd14c733b56b59f49e7d00e3ca4329c0aa1abed10fc4"}
Oct 30 19:33:25.195071 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.195048    4942 scope.go:122] "RemoveContainer" containerID="b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"
Oct 30 19:33:25.204164 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.204144    4942 scope.go:122] "RemoveContainer" containerID="5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"
Oct 30 19:33:25.211793 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.211775    4942 scope.go:122] "RemoveContainer" containerID="3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"
Oct 30 19:33:25.218649 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.218628    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:33:25.219412 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.219394    4942 scope.go:122] "RemoveContainer" containerID="b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"
Oct 30 19:33:25.219685 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:33:25.219664    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec\": container with ID starting with b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec not found: ID does not exist" containerID="b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"
Oct 30 19:33:25.219752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.219694    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec"} err="failed to get container status \"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec\": rpc error: code = NotFound desc = could not find container \"b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec\": container with ID starting with b9e0f39b9d7cb6055bc629d1fdeadadcb405a8089ffc2452bccc30b5c7fe72ec not found: ID does not exist"
Oct 30 19:33:25.219752 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.219713    4942 scope.go:122] "RemoveContainer" containerID="5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"
Oct 30 19:33:25.219941 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:33:25.219923    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9\": container with ID starting with 5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9 not found: ID does not exist" containerID="5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"
Oct 30 19:33:25.219984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.219944    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9"} err="failed to get container status \"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9\": rpc error: code = NotFound desc = could not find container \"5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9\": container with ID starting with 5844d5a62410bcc1a9e2a4a355c6ad5b180d887902d1ec425861da3fd13700d9 not found: ID does not exist"
Oct 30 19:33:25.219984 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.219962    4942 scope.go:122] "RemoveContainer" containerID="3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"
Oct 30 19:33:25.220176 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:33:25.220157    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef\": container with ID starting with 3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef not found: ID does not exist" containerID="3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"
Oct 30 19:33:25.220223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.220183    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef"} err="failed to get container status \"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef\": rpc error: code = NotFound desc = could not find container \"3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef\": container with ID starting with 3cd5d6d242cc217c7cc54c438c41f96228627011df10a0e7ce9c475c02c7a3ef not found: ID does not exist"
Oct 30 19:33:25.222986 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.222963    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-predictor-85767878c8-hrtpd"]
Oct 30 19:33:25.548137 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.548054    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" path="/var/lib/kubelet/pods/fb6d72a2-37af-4460-8990-696343fef0ae/volumes"
Oct 30 19:33:25.651341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651310    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:33:25.651670 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651657    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container"
Oct 30 19:33:25.651717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651672    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container"
Oct 30 19:33:25.651717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651693    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kube-rbac-proxy"
Oct 30 19:33:25.651717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651698    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kube-rbac-proxy"
Oct 30 19:33:25.651717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651708    4942 cpu_manager.go:414] "containerMap: removing container" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="storage-initializer"
Oct 30 19:33:25.651717 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651713    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="storage-initializer"
Oct 30 19:33:25.651906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651777    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kserve-container"
Oct 30 19:33:25.651906 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.651789    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6d72a2-37af-4460-8990-696343fef0ae" containerName="kube-rbac-proxy"
Oct 30 19:33:25.656694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.656668    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.660962 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.660929    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:33:25.661110 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.661047    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:33:25.661110 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.661058    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:33:25.661194 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.661111    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:33:25.663629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.663598    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:33:25.665837 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.665813    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:33:25.761257 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.761235    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:33:25.761365 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.761312    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Oct 30 19:33:25.763612 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.763590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.763725 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.763624    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.763768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.763724    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.763768 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.763755    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvq6\" (UniqueName: \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.763859 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.763782    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864318    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864360    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvq6\" (UniqueName: \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864386 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864392    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864417    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864633 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864437    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.864834 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.864815    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.865305 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.865284    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.865360 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.865284    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.866771 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.866750    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.877467 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.877440    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvq6\" (UniqueName: \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\") pod \"isvc-paddle-runtime-predictor-6768d897c4-wn225\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:25.974838 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:25.974808    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:26.253982 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:26.253952    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:33:26.256680 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:33:26.256653    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7552e3e1_1abf_4b2d_9e8a_26bcf6a5390a.slice/crio-e6be3106d3b227cd81c6c53dbb2ec9e3c8de61eff75b53f847b2c719654eb925 WatchSource:0}: Error finding container e6be3106d3b227cd81c6c53dbb2ec9e3c8de61eff75b53f847b2c719654eb925: Status 404 returned error can't find the container with id e6be3106d3b227cd81c6c53dbb2ec9e3c8de61eff75b53f847b2c719654eb925
Oct 30 19:33:26.724405 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:33:26.724367    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:33:27.205862 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:27.205822    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerStarted","Data":"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"}
Oct 30 19:33:27.206357 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:27.205897    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerStarted","Data":"e6be3106d3b227cd81c6c53dbb2ec9e3c8de61eff75b53f847b2c719654eb925"}
Oct 30 19:33:32.232860 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:32.232824    4942 generic.go:356] "Generic (PLEG): container finished" podID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerID="0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7" exitCode=0
Oct 30 19:33:32.233235 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:32.232899    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerDied","Data":"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"}
Oct 30 19:33:33.239681 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:33.239641    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerStarted","Data":"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"}
Oct 30 19:33:33.240098 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:33.239736    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerStarted","Data":"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"}
Oct 30 19:33:33.240098 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:33.240040    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:34.244154 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:34.244113    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:34.245367 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:34.245335    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:33:35.248663 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:35.248612    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:33:40.253203 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:40.253164    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:33:40.253806 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:40.253777    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:33:40.280395 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:40.280334    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podStartSLOduration=9.961166848 podStartE2EDuration="15.280319217s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.319152369s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:33.266530358 +0000 UTC m=+3399.844626049" watchObservedRunningTime="2026-10-30 19:33:40.280319217 +0000 UTC m=+3406.858414903"
Oct 30 19:33:43.129219 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:43.129181    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:50.124744 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.124704    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:33:50.125175 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.125046    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kserve-container" containerID="cri-o://fa324df5718ecb609fa870c088fea82efdacba064f659ee0eb7bcbab27d771c6" gracePeriod=30
Oct 30 19:33:50.125438 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.125415    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kube-rbac-proxy" containerID="cri-o://39fb7bb79b099161c8458a541b448163a631533d03f1fcc0a5179ff4bf3cb8a9" gracePeriod=30
Oct 30 19:33:50.253842 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.253792    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:33:50.315946 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.315912    4942 generic.go:356] "Generic (PLEG): container finished" podID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerID="39fb7bb79b099161c8458a541b448163a631533d03f1fcc0a5179ff4bf3cb8a9" exitCode=0
Oct 30 19:33:50.316182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:50.315980    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerDied","Data":"39fb7bb79b099161c8458a541b448163a631533d03f1fcc0a5179ff4bf3cb8a9"}
Oct 30 19:33:51.325576 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.325531    4942 generic.go:356] "Generic (PLEG): container finished" podID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerID="fa324df5718ecb609fa870c088fea82efdacba064f659ee0eb7bcbab27d771c6" exitCode=0
Oct 30 19:33:51.326061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.325664    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerDied","Data":"fa324df5718ecb609fa870c088fea82efdacba064f659ee0eb7bcbab27d771c6"}
Oct 30 19:33:51.386107 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.385967    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:51.478703 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.478610    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\") pod \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") "
Oct 30 19:33:51.478864 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.478717    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\") pod \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") "
Oct 30 19:33:51.478864 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.478765    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\" (UniqueName: \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\") pod \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") "
Oct 30 19:33:51.478864 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.478803    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\") pod \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") "
Oct 30 19:33:51.478864 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.478853    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\") pod \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\" (UID: \"6da72633-02d6-47a7-a8bb-10c6827bfa3b\") "
Oct 30 19:33:51.479119 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.479093    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config" pod "6da72633-02d6-47a7-a8bb-10c6827bfa3b" (UID: "6da72633-02d6-47a7-a8bb-10c6827bfa3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:51.479241 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.479160    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location" pod "6da72633-02d6-47a7-a8bb-10c6827bfa3b" (UID: "6da72633-02d6-47a7-a8bb-10c6827bfa3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:51.479327 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.479303    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert" pod "6da72633-02d6-47a7-a8bb-10c6827bfa3b" (UID: "6da72633-02d6-47a7-a8bb-10c6827bfa3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:51.481160 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.481132    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j" pod "6da72633-02d6-47a7-a8bb-10c6827bfa3b" (UID: "6da72633-02d6-47a7-a8bb-10c6827bfa3b"). InnerVolumeSpecName "kube-api-access-2h56j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:51.481282 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.481236    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls" pod "6da72633-02d6-47a7-a8bb-10c6827bfa3b" (UID: "6da72633-02d6-47a7-a8bb-10c6827bfa3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:51.580061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.579994    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.580061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.580056    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.580061 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.580067    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h56j\" (UniqueName: \"kubernetes.io/projected/6da72633-02d6-47a7-a8bb-10c6827bfa3b-kube-api-access-2h56j\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.580301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.580077    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da72633-02d6-47a7-a8bb-10c6827bfa3b-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.580301 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:51.580087    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da72633-02d6-47a7-a8bb-10c6827bfa3b-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.332878 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.332841    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd" event={"ID":"6da72633-02d6-47a7-a8bb-10c6827bfa3b","Type":"ContainerDied","Data":"8ba310a76e77b221ffb3f35fb6e16ce283c97447ea3c314a86e8f494ebda72b0"}
Oct 30 19:33:52.332878 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.332866    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"
Oct 30 19:33:52.333483 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.332911    4942 scope.go:122] "RemoveContainer" containerID="39fb7bb79b099161c8458a541b448163a631533d03f1fcc0a5179ff4bf3cb8a9"
Oct 30 19:33:52.341616 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.341591    4942 scope.go:122] "RemoveContainer" containerID="fa324df5718ecb609fa870c088fea82efdacba064f659ee0eb7bcbab27d771c6"
Oct 30 19:33:52.350226 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.350197    4942 scope.go:122] "RemoveContainer" containerID="0dda07aa70e245503df4ee1d1ab8c75710900095b1b4965d8f1b0fcdae50fac2"
Oct 30 19:33:52.356666 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.356628    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:33:52.360243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:52.360209    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-lightgbm-v2-runtime-predictor-75dddfb46c-9nzrd"]
Oct 30 19:33:53.542453 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:33:53.542415    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" path="/var/lib/kubelet/pods/6da72633-02d6-47a7-a8bb-10c6827bfa3b/volumes"
Oct 30 19:33:56.725398 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:33:56.725360    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:34:00.254309 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:00.254268    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:34:10.253844 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:10.253797    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:34:20.254053 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:20.253995    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:34:26.727074 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:34:26.727031    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:34:27.414997 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.414953    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:34:27.415318 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415304    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="storage-initializer"
Oct 30 19:34:27.415362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415319    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="storage-initializer"
Oct 30 19:34:27.415362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415330    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kube-rbac-proxy"
Oct 30 19:34:27.415362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415335    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kube-rbac-proxy"
Oct 30 19:34:27.415362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415345    4942 cpu_manager.go:414] "containerMap: removing container" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kserve-container"
Oct 30 19:34:27.415362 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415350    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kserve-container"
Oct 30 19:34:27.415514 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415430    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kube-rbac-proxy"
Oct 30 19:34:27.415514 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.415438    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da72633-02d6-47a7-a8bb-10c6827bfa3b" containerName="kserve-container"
Oct 30 19:34:27.418923 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.418874    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.423617 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.423577    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:34:27.424913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.424864    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:27.425336 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.425310    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:34:27.425466 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.425331    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:34:27.425466 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.425315    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:34:27.425466 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.425374    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 30 19:34:27.425466 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.425418    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:27.434531 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.434496    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:34:27.489233 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.489188    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.489233 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.489237    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hmr\" (UniqueName: \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.489501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.489416    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.489501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.489477    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.489609 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.489542    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590173 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590132    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590173 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590174    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590199    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590220    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hmr\" (UniqueName: \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590303    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590719 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590692    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590939 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590916    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.590981 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.590934    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.592785 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.592761    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.604243 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.604203    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hmr\" (UniqueName: \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\") pod \"isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") " pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:27.741397 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:27.741343    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:28.025758 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:28.025669    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:34:28.029491 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:34:28.029459    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ff9467_a1d8_411a_bd76_b036cd77c1ff.slice/crio-180d4fa7df6e0cfb3cbe059da8d10e284314c9f90ea490adee85482459bdaeb4 WatchSource:0}: Error finding container 180d4fa7df6e0cfb3cbe059da8d10e284314c9f90ea490adee85482459bdaeb4: Status 404 returned error can't find the container with id 180d4fa7df6e0cfb3cbe059da8d10e284314c9f90ea490adee85482459bdaeb4
Oct 30 19:34:28.503581 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:28.503537    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerStarted","Data":"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"}
Oct 30 19:34:28.503774 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:28.503617    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerStarted","Data":"180d4fa7df6e0cfb3cbe059da8d10e284314c9f90ea490adee85482459bdaeb4"}
Oct 30 19:34:30.254259 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:30.254220    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:34:33.528445 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:33.528403    4942 generic.go:356] "Generic (PLEG): container finished" podID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerID="01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2" exitCode=0
Oct 30 19:34:33.528881 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:33.528483    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerDied","Data":"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"}
Oct 30 19:34:34.534571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:34.534532    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerStarted","Data":"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"}
Oct 30 19:34:34.535075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:34.534603    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerStarted","Data":"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"}
Oct 30 19:34:34.535075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:34.534806    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:34.535075 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:34.534860    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:34.562802 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:34.562747    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" podStartSLOduration=2.582686745 podStartE2EDuration="7.562731795s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.98004505s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:34.560908808 +0000 UTC m=+3461.139004499" watchObservedRunningTime="2026-10-30 19:34:34.562731795 +0000 UTC m=+3461.140827487"
Oct 30 19:34:37.589893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:37.589845    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:34:37.590313 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:37.590191    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" containerID="cri-o://a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b" gracePeriod=30
Oct 30 19:34:37.590665 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:37.590636    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kube-rbac-proxy" containerID="cri-o://846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b" gracePeriod=30
Oct 30 19:34:38.558494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:38.558446    4942 generic.go:356] "Generic (PLEG): container finished" podID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerID="846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b" exitCode=0
Oct 30 19:34:38.558723 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:38.558514    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerDied","Data":"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"}
Oct 30 19:34:40.249079 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:40.249026    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.83:8643/healthz\": dial tcp 10.132.0.83:8643: connect: connection refused"
Oct 30 19:34:40.253814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:40.253775    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 19:34:40.546206 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:40.546109    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:34:40.935994 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:40.935966    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:34:41.008118 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008091    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\") pod \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") "
Oct 30 19:34:41.008256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008122    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\") pod \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") "
Oct 30 19:34:41.008256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008145    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\" (UniqueName: \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\") pod \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") "
Oct 30 19:34:41.008256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008206    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\") pod \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") "
Oct 30 19:34:41.008256 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008226    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\") pod \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\" (UID: \"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a\") "
Oct 30 19:34:41.008507 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008477    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config" pod "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" (UID: "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:41.008659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.008633    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert" pod "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" (UID: "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:41.010889 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.010864    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls" pod "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" (UID: "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:41.010971 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.010903    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6" pod "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" (UID: "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a"). InnerVolumeSpecName "kube-api-access-pcvq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:41.018085 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.018061    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location" pod "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" (UID: "7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.108854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.108786    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.108854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.108811    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.108854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.108821    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.108854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.108832    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.108854 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.108841    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvq6\" (UniqueName: \"kubernetes.io/projected/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a-kube-api-access-pcvq6\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.575195 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.575166    4942 generic.go:356] "Generic (PLEG): container finished" podID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerID="a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b" exitCode=0
Oct 30 19:34:41.575525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.575221    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"
Oct 30 19:34:41.575525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.575227    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerDied","Data":"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"}
Oct 30 19:34:41.575525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.575355    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225" event={"ID":"7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a","Type":"ContainerDied","Data":"e6be3106d3b227cd81c6c53dbb2ec9e3c8de61eff75b53f847b2c719654eb925"}
Oct 30 19:34:41.575525 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.575388    4942 scope.go:122] "RemoveContainer" containerID="846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"
Oct 30 19:34:41.583712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.583694    4942 scope.go:122] "RemoveContainer" containerID="a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"
Oct 30 19:34:41.591153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.591131    4942 scope.go:122] "RemoveContainer" containerID="0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"
Oct 30 19:34:41.595381 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.595361    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:34:41.599624 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.599609    4942 scope.go:122] "RemoveContainer" containerID="846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"
Oct 30 19:34:41.599937 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:34:41.599916    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b\": container with ID starting with 846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b not found: ID does not exist" containerID="846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"
Oct 30 19:34:41.599987 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.599957    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b"} err="failed to get container status \"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b\": rpc error: code = NotFound desc = could not find container \"846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b\": container with ID starting with 846add26dacff94a23ffa829691c6235e25a1c87250ea2dc4551ea8be811c57b not found: ID does not exist"
Oct 30 19:34:41.599987 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.599978    4942 scope.go:122] "RemoveContainer" containerID="a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"
Oct 30 19:34:41.600293 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:34:41.600273    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b\": container with ID starting with a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b not found: ID does not exist" containerID="a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"
Oct 30 19:34:41.600358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.600297    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b"} err="failed to get container status \"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b\": rpc error: code = NotFound desc = could not find container \"a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b\": container with ID starting with a107772d3001829999aeb9e184404a696c4b9ed1006fdd22b650bab9bbaa528b not found: ID does not exist"
Oct 30 19:34:41.600358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.600312    4942 scope.go:122] "RemoveContainer" containerID="0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"
Oct 30 19:34:41.600577 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:34:41.600559    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7\": container with ID starting with 0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7 not found: ID does not exist" containerID="0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"
Oct 30 19:34:41.600625 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.600581    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7"} err="failed to get container status \"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7\": rpc error: code = NotFound desc = could not find container \"0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7\": container with ID starting with 0b6522027589c5f603a57b2254f8230ea068b5b32856d293d40a44f4f3d58ac7 not found: ID does not exist"
Oct 30 19:34:41.601153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:41.601135    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-runtime-predictor-6768d897c4-wn225"]
Oct 30 19:34:43.032965 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.032931    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:34:43.033370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033324    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="storage-initializer"
Oct 30 19:34:43.033370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033337    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="storage-initializer"
Oct 30 19:34:43.033370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033359    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kube-rbac-proxy"
Oct 30 19:34:43.033370 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033364    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kube-rbac-proxy"
Oct 30 19:34:43.033502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033373    4942 cpu_manager.go:414] "containerMap: removing container" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container"
Oct 30 19:34:43.033502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033378    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container"
Oct 30 19:34:43.033502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033443    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kube-rbac-proxy"
Oct 30 19:34:43.033502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.033451    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" containerName="kserve-container"
Oct 30 19:34:43.039353 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.039308    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.051136 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051103    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:43.051136 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051125    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:43.051354 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051116    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 19:34:43.051916 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051894    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Oct 30 19:34:43.052101 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051896    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:34:43.052202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051932    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:34:43.052202 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.051968    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:34:43.052851 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.052833    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:34:43.127602 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.127555    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbjt\" (UniqueName: \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.127783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.127619    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.127783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.127656    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.127783 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.127702    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.127893 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.127791    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228210    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbjt\" (UniqueName: \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228246 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228250    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228276    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228316    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228502 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228380    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.228766 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.228707    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.229066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.229046    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.229125 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.229074    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.231162 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.231131    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.241424 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.241386    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbjt\" (UniqueName: \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\") pod \"isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") " pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.359272 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.359175    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:43.543028 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.542975    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a" path="/var/lib/kubelet/pods/7552e3e1-1abf-4b2d-9e8a-26bcf6a5390a/volumes"
Oct 30 19:34:43.644833 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:43.644747    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:34:43.648190 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:34:43.648154    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb06e1f7_4930_4910_b1db_73767dc63300.slice/crio-f9fdde9071bd3ce431db207b0eeaa00f3fdc9a6317df7c374eae010f2b8df45d WatchSource:0}: Error finding container f9fdde9071bd3ce431db207b0eeaa00f3fdc9a6317df7c374eae010f2b8df45d: Status 404 returned error can't find the container with id f9fdde9071bd3ce431db207b0eeaa00f3fdc9a6317df7c374eae010f2b8df45d
Oct 30 19:34:44.590292 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:44.590255    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerStarted","Data":"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"}
Oct 30 19:34:44.590741 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:44.590327    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerStarted","Data":"f9fdde9071bd3ce431db207b0eeaa00f3fdc9a6317df7c374eae010f2b8df45d"}
Oct 30 19:34:49.615573 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:49.615533    4942 generic.go:356] "Generic (PLEG): container finished" podID="bb06e1f7-4930-4910-b1db-73767dc63300" containerID="1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169" exitCode=0
Oct 30 19:34:49.615993 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:49.615591    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerDied","Data":"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"}
Oct 30 19:34:50.623115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.623071    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerStarted","Data":"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"}
Oct 30 19:34:50.623572 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.623141    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerStarted","Data":"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"}
Oct 30 19:34:50.623572 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.623477    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:50.623664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.623619    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:50.625325 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.625287    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:34:50.651247 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:50.651180    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podStartSLOduration=2.532704823 podStartE2EDuration="7.651162665s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.118457842s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:50.649762265 +0000 UTC m=+3477.227857994" watchObservedRunningTime="2026-10-30 19:34:50.651162665 +0000 UTC m=+3477.229258358"
Oct 30 19:34:51.628487 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:51.628433    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:34:56.633416 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:56.633383    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:34:56.634029 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:34:56.633979    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:34:56.725370 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:34:56.725325    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:35:06.634306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:06.634265    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:35:10.550479 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:10.550440    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:35:16.633856 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:16.633801    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:35:17.713987 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:17.713952    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:35:17.714441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:17.714187    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kserve-container" containerID="cri-o://c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634" gracePeriod=30
Oct 30 19:35:17.714441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:17.714236    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kube-rbac-proxy" containerID="cri-o://5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30" gracePeriod=30
Oct 30 19:35:18.762251 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:18.762216    4942 generic.go:356] "Generic (PLEG): container finished" podID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerID="5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30" exitCode=0
Oct 30 19:35:18.762652 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:18.762302    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerDied","Data":"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"}
Oct 30 19:35:19.070503 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.070477    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:35:19.131749 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.131719    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\" (UniqueName: \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\") pod \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") "
Oct 30 19:35:19.131949 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.131780    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\") pod \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") "
Oct 30 19:35:19.131949 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.131809    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\") pod \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") "
Oct 30 19:35:19.131949 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.131841    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\") pod \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") "
Oct 30 19:35:19.132264 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.132238    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config" pod "11ff9467-a1d8-411a-bd76-b036cd77c1ff" (UID: "11ff9467-a1d8-411a-bd76-b036cd77c1ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:19.132324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.132252    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location" pod "11ff9467-a1d8-411a-bd76-b036cd77c1ff" (UID: "11ff9467-a1d8-411a-bd76-b036cd77c1ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:19.132324 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.132269    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert" pod "11ff9467-a1d8-411a-bd76-b036cd77c1ff" (UID: "11ff9467-a1d8-411a-bd76-b036cd77c1ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:19.133737 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.133717    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls" pod "11ff9467-a1d8-411a-bd76-b036cd77c1ff" (UID: "11ff9467-a1d8-411a-bd76-b036cd77c1ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:19.232925 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.232894    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\" (UniqueName: \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\") pod \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\" (UID: \"11ff9467-a1d8-411a-bd76-b036cd77c1ff\") "
Oct 30 19:35:19.233097 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.233086    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11ff9467-a1d8-411a-bd76-b036cd77c1ff-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.233141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.233099    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.233141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.233109    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.233141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.233118    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9467-a1d8-411a-bd76-b036cd77c1ff-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.234896 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.234875    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr" pod "11ff9467-a1d8-411a-bd76-b036cd77c1ff" (UID: "11ff9467-a1d8-411a-bd76-b036cd77c1ff"). InnerVolumeSpecName "kube-api-access-j8hmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:19.334538 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.334450    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hmr\" (UniqueName: \"kubernetes.io/projected/11ff9467-a1d8-411a-bd76-b036cd77c1ff-kube-api-access-j8hmr\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.768472 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.768428    4942 generic.go:356] "Generic (PLEG): container finished" podID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerID="c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634" exitCode=0
Oct 30 19:35:19.768896 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.768504    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"
Oct 30 19:35:19.768896 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.768499    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerDied","Data":"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"}
Oct 30 19:35:19.768896 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.768572    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg" event={"ID":"11ff9467-a1d8-411a-bd76-b036cd77c1ff","Type":"ContainerDied","Data":"180d4fa7df6e0cfb3cbe059da8d10e284314c9f90ea490adee85482459bdaeb4"}
Oct 30 19:35:19.768896 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.768605    4942 scope.go:122] "RemoveContainer" containerID="5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"
Oct 30 19:35:19.779174 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.779136    4942 scope.go:122] "RemoveContainer" containerID="c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"
Oct 30 19:35:19.788155 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.788134    4942 scope.go:122] "RemoveContainer" containerID="01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"
Oct 30 19:35:19.790481 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.790456    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:35:19.795120 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.795094    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-mlflow-v2-runtime-predictor-766864c8c6-h5rbg"]
Oct 30 19:35:19.796945 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.796929    4942 scope.go:122] "RemoveContainer" containerID="5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"
Oct 30 19:35:19.797261 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:19.797243    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30\": container with ID starting with 5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30 not found: ID does not exist" containerID="5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"
Oct 30 19:35:19.797352 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.797272    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30"} err="failed to get container status \"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30\": rpc error: code = NotFound desc = could not find container \"5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30\": container with ID starting with 5f3b98057d6a28fa4dee93bd25702b421f6df3abfe53e0d30232119d717ebb30 not found: ID does not exist"
Oct 30 19:35:19.797352 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.797299    4942 scope.go:122] "RemoveContainer" containerID="c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"
Oct 30 19:35:19.797548 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:19.797530    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634\": container with ID starting with c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634 not found: ID does not exist" containerID="c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"
Oct 30 19:35:19.797613 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.797558    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634"} err="failed to get container status \"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634\": rpc error: code = NotFound desc = could not find container \"c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634\": container with ID starting with c77590a87274e8ea5b02fd06b940b85c23ee5e4fc25904090ef138b0ac2c7634 not found: ID does not exist"
Oct 30 19:35:19.797613 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.797579    4942 scope.go:122] "RemoveContainer" containerID="01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"
Oct 30 19:35:19.797789 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:19.797773    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2\": container with ID starting with 01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2 not found: ID does not exist" containerID="01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"
Oct 30 19:35:19.797845 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:19.797796    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2"} err="failed to get container status \"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2\": rpc error: code = NotFound desc = could not find container \"01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2\": container with ID starting with 01b0ef4c60b7c27c9443e5f44027ce94ea2f8c1147d7c71be392ae0db8964ea2 not found: ID does not exist"
Oct 30 19:35:21.542483 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:21.542441    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" path="/var/lib/kubelet/pods/11ff9467-a1d8-411a-bd76-b036cd77c1ff/volumes"
Oct 30 19:35:26.634058 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:26.634000    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:35:26.723430 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:26.723394    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:35:36.634183 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:36.634144    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:35:46.634979 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:46.634945    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:35:55.112592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:55.112552    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:35:55.113115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:55.112807    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" containerID="cri-o://1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2" gracePeriod=30
Oct 30 19:35:55.113115 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:55.112881    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kube-rbac-proxy" containerID="cri-o://d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5" gracePeriod=30
Oct 30 19:35:55.922611 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:55.922571    4942 generic.go:356] "Generic (PLEG): container finished" podID="bb06e1f7-4930-4910-b1db-73767dc63300" containerID="d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5" exitCode=0
Oct 30 19:35:55.922797 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:55.922625    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerDied","Data":"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"}
Oct 30 19:35:56.629259 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:56.629208    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.85:8643/healthz\": dial tcp 10.132.0.85:8643: connect: connection refused"
Oct 30 19:35:56.634686 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:56.634647    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 30 19:35:56.726280 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:56.726238    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:35:58.660037 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.659972    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:35:58.763483 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763382    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\") pod \"bb06e1f7-4930-4910-b1db-73767dc63300\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") "
Oct 30 19:35:58.763483 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763438    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\") pod \"bb06e1f7-4930-4910-b1db-73767dc63300\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") "
Oct 30 19:35:58.763730 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763496    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\") pod \"bb06e1f7-4930-4910-b1db-73767dc63300\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") "
Oct 30 19:35:58.763730 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763517    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\" (UniqueName: \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\") pod \"bb06e1f7-4930-4910-b1db-73767dc63300\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") "
Oct 30 19:35:58.763730 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763663    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\") pod \"bb06e1f7-4930-4910-b1db-73767dc63300\" (UID: \"bb06e1f7-4930-4910-b1db-73767dc63300\") "
Oct 30 19:35:58.763908 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763795    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config" pod "bb06e1f7-4930-4910-b1db-73767dc63300" (UID: "bb06e1f7-4930-4910-b1db-73767dc63300"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:58.763990 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763965    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:58.764072 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.763977    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert" pod "bb06e1f7-4930-4910-b1db-73767dc63300" (UID: "bb06e1f7-4930-4910-b1db-73767dc63300"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:58.765935 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.765896    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls" pod "bb06e1f7-4930-4910-b1db-73767dc63300" (UID: "bb06e1f7-4930-4910-b1db-73767dc63300"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:58.766054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.766001    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt" pod "bb06e1f7-4930-4910-b1db-73767dc63300" (UID: "bb06e1f7-4930-4910-b1db-73767dc63300"). InnerVolumeSpecName "kube-api-access-8cbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:58.774247 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.774203    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location" pod "bb06e1f7-4930-4910-b1db-73767dc63300" (UID: "bb06e1f7-4930-4910-b1db-73767dc63300"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:58.864724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.864677    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb06e1f7-4930-4910-b1db-73767dc63300-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:58.864724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.864717    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb06e1f7-4930-4910-b1db-73767dc63300-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:58.864724 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.864728    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb06e1f7-4930-4910-b1db-73767dc63300-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:58.864968 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.864738    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbjt\" (UniqueName: \"kubernetes.io/projected/bb06e1f7-4930-4910-b1db-73767dc63300-kube-api-access-8cbjt\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:58.938788 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.938749    4942 generic.go:356] "Generic (PLEG): container finished" podID="bb06e1f7-4930-4910-b1db-73767dc63300" containerID="1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2" exitCode=0
Oct 30 19:35:58.938934 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.938811    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"
Oct 30 19:35:58.938934 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.938837    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerDied","Data":"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"}
Oct 30 19:35:58.938934 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.938895    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw" event={"ID":"bb06e1f7-4930-4910-b1db-73767dc63300","Type":"ContainerDied","Data":"f9fdde9071bd3ce431db207b0eeaa00f3fdc9a6317df7c374eae010f2b8df45d"}
Oct 30 19:35:58.938934 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.938924    4942 scope.go:122] "RemoveContainer" containerID="d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"
Oct 30 19:35:58.948688 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.948664    4942 scope.go:122] "RemoveContainer" containerID="1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"
Oct 30 19:35:58.958072 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.958047    4942 scope.go:122] "RemoveContainer" containerID="1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"
Oct 30 19:35:58.964417 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.964380    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:35:58.969514 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.969482    4942 scope.go:122] "RemoveContainer" containerID="d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"
Oct 30 19:35:58.969924 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:58.969899    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5\": container with ID starting with d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5 not found: ID does not exist" containerID="d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"
Oct 30 19:35:58.970032 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.969938    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5"} err="failed to get container status \"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5\": rpc error: code = NotFound desc = could not find container \"d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5\": container with ID starting with d150332ec24929919e57289a78f0d64c5a458ca21a977179f883ca03aef73be5 not found: ID does not exist"
Oct 30 19:35:58.970032 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.969976    4942 scope.go:122] "RemoveContainer" containerID="1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"
Oct 30 19:35:58.970376 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:58.970353    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2\": container with ID starting with 1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2 not found: ID does not exist" containerID="1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"
Oct 30 19:35:58.970441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.970383    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2"} err="failed to get container status \"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2\": rpc error: code = NotFound desc = could not find container \"1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2\": container with ID starting with 1d8115126795868f11b5c2d8d645abcf1b786393099b5716f1c3812e50fb45c2 not found: ID does not exist"
Oct 30 19:35:58.970441 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.970400    4942 scope.go:122] "RemoveContainer" containerID="1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"
Oct 30 19:35:58.970671 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:35:58.970651    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169\": container with ID starting with 1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169 not found: ID does not exist" containerID="1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"
Oct 30 19:35:58.970727 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.970680    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169"} err="failed to get container status \"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169\": rpc error: code = NotFound desc = could not find container \"1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169\": container with ID starting with 1a52f6083e1bd2b2f67156467c94601fe85dce34651c5eb88befea49623a7169 not found: ID does not exist"
Oct 30 19:35:58.975358 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:58.975321    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-paddle-v2-kserve-predictor-79cdb6b66b-4tlpw"]
Oct 30 19:35:59.541803 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:35:59.541763    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" path="/var/lib/kubelet/pods/bb06e1f7-4930-4910-b1db-73767dc63300/volumes"
Oct 30 19:36:00.483691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.483649    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484045    4942 cpu_manager.go:414] "containerMap: removing container" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484058    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484068    4942 cpu_manager.go:414] "containerMap: removing container" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kube-rbac-proxy"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484074    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kube-rbac-proxy"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484098    4942 cpu_manager.go:414] "containerMap: removing container" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="storage-initializer"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484107    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="storage-initializer"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484117    4942 cpu_manager.go:414] "containerMap: removing container" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kserve-container"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484125    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kserve-container"
Oct 30 19:36:00.484133 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484137    4942 cpu_manager.go:414] "containerMap: removing container" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="storage-initializer"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484145    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="storage-initializer"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484158    4942 cpu_manager.go:414] "containerMap: removing container" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kube-rbac-proxy"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484165    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kube-rbac-proxy"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484247    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kserve-container"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484258    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb06e1f7-4930-4910-b1db-73767dc63300" containerName="kube-rbac-proxy"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484271    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kserve-container"
Oct 30 19:36:00.484499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.484282    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ff9467-a1d8-411a-bd76-b036cd77c1ff" containerName="kube-rbac-proxy"
Oct 30 19:36:00.489678 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.489632    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.498673 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.498643    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Oct 30 19:36:00.502675 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.502640    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:36:00.503659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.503630    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:36:00.503659 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.503657    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"openshift-service-ca.crt\""
Oct 30 19:36:00.503900 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.503657    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:36:00.503900 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.503693    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"kube-root-ca.crt\""
Oct 30 19:36:00.579911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.579869    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.579911 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.579917    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgg9c\" (UniqueName: \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.580157 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.579940    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.580157 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.580090    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.580238 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.580203    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.597099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.597068    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"default-dockercfg-ls7pb\""
Oct 30 19:36:00.597099 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.597068    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c83bbf5b66\"/\"seaweedfs-s3-creds\""
Oct 30 19:36:00.680858 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.680810    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.680858 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.680856    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgg9c\" (UniqueName: \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681109 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.680878    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681109 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.680915    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681109 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.680963    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681508 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.681483    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681786 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.681754    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.681836 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.681759    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.683959 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.683919    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.694312 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.694277    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgg9c\" (UniqueName: \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\") pod \"isvc-pmml-runtime-predictor-654bc497b5-q68mr\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") " pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:00.809931 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:00.809829    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:01.096169 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:01.096030    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:36:01.099401 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:36:01.099365    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cf9a32_67f7_4484_a023_ee52c4050113.slice/crio-9e59bd62d279ae9ac2a6b0c070aac679078aaa5c4426f3123a649945bcad79c9 WatchSource:0}: Error finding container 9e59bd62d279ae9ac2a6b0c070aac679078aaa5c4426f3123a649945bcad79c9: Status 404 returned error can't find the container with id 9e59bd62d279ae9ac2a6b0c070aac679078aaa5c4426f3123a649945bcad79c9
Oct 30 19:36:01.954660 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:01.954619    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerStarted","Data":"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"}
Oct 30 19:36:01.955153 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:01.954684    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerStarted","Data":"9e59bd62d279ae9ac2a6b0c070aac679078aaa5c4426f3123a649945bcad79c9"}
Oct 30 19:36:05.974147 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:05.974102    4942 generic.go:356] "Generic (PLEG): container finished" podID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerID="efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73" exitCode=0
Oct 30 19:36:05.974598 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:05.974198    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerDied","Data":"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"}
Oct 30 19:36:05.975966 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:05.975938    4942 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:36:14.018560 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.018519    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerStarted","Data":"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"}
Oct 30 19:36:14.019089 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.018596    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerStarted","Data":"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"}
Oct 30 19:36:14.019089 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.018936    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:14.019089 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.019035    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:14.020254 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.020223    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:14.047816 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:14.047752    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podStartSLOduration=2.061350605 podStartE2EDuration="14.047736723s" totalImagesPullingTime="7.88878405s" totalInitContainerRuntime="4.097602068s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:14.04538604 +0000 UTC m=+3560.623481744" watchObservedRunningTime="2026-10-30 19:36:14.047736723 +0000 UTC m=+3560.625832415"
Oct 30 19:36:15.023024 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:15.022981    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:20.027910 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:20.027881    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:36:20.028426 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:20.028399    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:26.724570 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:36:26.724527    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:36:30.029164 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:30.029118    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:40.028428 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:40.028391    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:50.029236 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:50.029196    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:36:53.862476 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:53.862369    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:36:53.905173 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:36:53.876997    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:36:56.724147 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:36:56.724111    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:37:00.028693 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:00.028650    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:10.028521 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:10.028478    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:13.337945 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.337908    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:37:13.341612 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.341594    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.345753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.345726    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-pmml-predictor-serving-cert\""
Oct 30 19:37:13.346052 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.345997    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"seaweedfs-s3-creds\""
Oct 30 19:37:13.347275 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.347254    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:37:13.347409 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.347273    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"kube-root-ca.crt\""
Oct 30 19:37:13.347462 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.347276    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Oct 30 19:37:13.347462 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.347285    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-3deaac58e4\"/\"default-dockercfg-fr56d\""
Oct 30 19:37:13.347691 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.347670    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-3deaac58e4\"/\"openshift-service-ca.crt\""
Oct 30 19:37:13.352927 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.352907    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:37:13.478096 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.478056    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.478260 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.478109    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.478260 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.478224    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z68\" (UniqueName: \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.478341 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.478270    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.478376 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.478361    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579040 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.578992    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579058    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579099    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579136    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579182 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579165    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z68\" (UniqueName: \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579509 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579489    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579779 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579759    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.579861 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.579782    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.581642 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.581623    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.592461 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.592410    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z68\" (UniqueName: \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\") pod \"isvc-pmml-predictor-7fcfc6d686-7k9qj\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") " pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.661571 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.661546    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:13.943270 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:13.943244    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:37:13.946104 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:37:13.946077    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94927a25_4147_41fe_b0f9_b803bc54e6e9.slice/crio-8c9278ac582b9fa5b03042f93662da97969b3130752ccbe0dc88d6dddc8c61cf WatchSource:0}: Error finding container 8c9278ac582b9fa5b03042f93662da97969b3130752ccbe0dc88d6dddc8c61cf: Status 404 returned error can't find the container with id 8c9278ac582b9fa5b03042f93662da97969b3130752ccbe0dc88d6dddc8c61cf
Oct 30 19:37:14.267558 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:14.267458    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerStarted","Data":"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"}
Oct 30 19:37:14.267732 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:14.267572    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerStarted","Data":"8c9278ac582b9fa5b03042f93662da97969b3130752ccbe0dc88d6dddc8c61cf"}
Oct 30 19:37:18.287913 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:18.287880    4942 generic.go:356] "Generic (PLEG): container finished" podID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerID="56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487" exitCode=0
Oct 30 19:37:18.288306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:18.287963    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerDied","Data":"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"}
Oct 30 19:37:19.293897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.293859    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerStarted","Data":"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"}
Oct 30 19:37:19.294402 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.293933    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerStarted","Data":"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"}
Oct 30 19:37:19.294402 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.294228    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:19.294402 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.294286    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:19.295557 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.295532    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:19.319373 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:19.319326    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podStartSLOduration=2.5626803369999998 podStartE2EDuration="6.319314775s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756634438s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:19.317665506 +0000 UTC m=+3625.895761200" watchObservedRunningTime="2026-10-30 19:37:19.319314775 +0000 UTC m=+3625.897410525"
Oct 30 19:37:20.028907 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:20.028871    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:20.298600 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:20.298511    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:25.303753 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:25.303717    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:37:25.304406 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:25.304373    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:26.724847 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:37:26.724810    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:37:30.028553 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:30.028507    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:35.305279 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:35.305203    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:40.028305 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:40.028267    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:44.537653 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:44.537621    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:37:45.305145 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:45.305103    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:52.534067 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:52.534035    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:37:52.534578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:52.534338    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" containerID="cri-o://94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a" gracePeriod=30
Oct 30 19:37:52.534578 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:52.534370    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kube-rbac-proxy" containerID="cri-o://8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f" gracePeriod=30
Oct 30 19:37:53.441340 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:53.441307    4942 generic.go:356] "Generic (PLEG): container finished" podID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerID="8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f" exitCode=0
Oct 30 19:37:53.441521 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:53.441382    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerDied","Data":"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"}
Oct 30 19:37:54.537859 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:54.537813    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 19:37:55.023295 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:55.023255    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.86:8643/healthz\": dial tcp 10.132.0.86:8643: connect: connection refused"
Oct 30 19:37:55.304635 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:55.304538    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:37:56.288764 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.288739    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:37:56.312048 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.311977    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\") pod \"c3cf9a32-67f7-4484-a023-ee52c4050113\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") "
Oct 30 19:37:56.312154 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312071    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\") pod \"c3cf9a32-67f7-4484-a023-ee52c4050113\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") "
Oct 30 19:37:56.312154 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312114    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\" (UniqueName: \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\") pod \"c3cf9a32-67f7-4484-a023-ee52c4050113\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") "
Oct 30 19:37:56.312154 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312141    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\") pod \"c3cf9a32-67f7-4484-a023-ee52c4050113\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") "
Oct 30 19:37:56.312284 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312164    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\") pod \"c3cf9a32-67f7-4484-a023-ee52c4050113\" (UID: \"c3cf9a32-67f7-4484-a023-ee52c4050113\") "
Oct 30 19:37:56.312342 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312294    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config" pod "c3cf9a32-67f7-4484-a023-ee52c4050113" (UID: "c3cf9a32-67f7-4484-a023-ee52c4050113"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:56.312501 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312397    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location" pod "c3cf9a32-67f7-4484-a023-ee52c4050113" (UID: "c3cf9a32-67f7-4484-a023-ee52c4050113"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:56.312654 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.312604    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert" pod "c3cf9a32-67f7-4484-a023-ee52c4050113" (UID: "c3cf9a32-67f7-4484-a023-ee52c4050113"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:56.314399 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.314370    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls" pod "c3cf9a32-67f7-4484-a023-ee52c4050113" (UID: "c3cf9a32-67f7-4484-a023-ee52c4050113"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:56.314544 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.314518    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c" pod "c3cf9a32-67f7-4484-a023-ee52c4050113" (UID: "c3cf9a32-67f7-4484-a023-ee52c4050113"). InnerVolumeSpecName "kube-api-access-dgg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:56.413432 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.413402    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cf9a32-67f7-4484-a023-ee52c4050113-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:56.413432 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.413427    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgg9c\" (UniqueName: \"kubernetes.io/projected/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-api-access-dgg9c\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:56.413432 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.413438    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:56.413623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.413447    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cf9a32-67f7-4484-a023-ee52c4050113-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:56.413623 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.413456    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cf9a32-67f7-4484-a023-ee52c4050113-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:56.457592 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.457559    4942 generic.go:356] "Generic (PLEG): container finished" podID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerID="94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a" exitCode=0
Oct 30 19:37:56.457709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.457630    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"
Oct 30 19:37:56.457709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.457640    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerDied","Data":"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"}
Oct 30 19:37:56.457709 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.457700    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr" event={"ID":"c3cf9a32-67f7-4484-a023-ee52c4050113","Type":"ContainerDied","Data":"9e59bd62d279ae9ac2a6b0c070aac679078aaa5c4426f3123a649945bcad79c9"}
Oct 30 19:37:56.457816 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.457726    4942 scope.go:122] "RemoveContainer" containerID="8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"
Oct 30 19:37:56.466404 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.466390    4942 scope.go:122] "RemoveContainer" containerID="94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"
Oct 30 19:37:56.474091 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.474072    4942 scope.go:122] "RemoveContainer" containerID="efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"
Oct 30 19:37:56.481562 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.481542    4942 scope.go:122] "RemoveContainer" containerID="8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"
Oct 30 19:37:56.481635 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.481618    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:37:56.481823 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:37:56.481804    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f\": container with ID starting with 8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f not found: ID does not exist" containerID="8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"
Oct 30 19:37:56.481874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.481832    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f"} err="failed to get container status \"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f\": rpc error: code = NotFound desc = could not find container \"8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f\": container with ID starting with 8a4a24598a01d1960c1bb46ab6494081c8012325fa767fbfda146b4c7ee6361f not found: ID does not exist"
Oct 30 19:37:56.481874 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.481848    4942 scope.go:122] "RemoveContainer" containerID="94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"
Oct 30 19:37:56.482105 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:37:56.482086    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a\": container with ID starting with 94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a not found: ID does not exist" containerID="94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"
Oct 30 19:37:56.482179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.482115    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a"} err="failed to get container status \"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a\": rpc error: code = NotFound desc = could not find container \"94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a\": container with ID starting with 94a00c5d02d1eabe029ff8c3027102b62b869998aa29c3ee8f00f0eb1c0c054a not found: ID does not exist"
Oct 30 19:37:56.482179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.482137    4942 scope.go:122] "RemoveContainer" containerID="efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"
Oct 30 19:37:56.482411 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:37:56.482394    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73\": container with ID starting with efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73 not found: ID does not exist" containerID="efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"
Oct 30 19:37:56.482450 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.482417    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73"} err="failed to get container status \"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73\": rpc error: code = NotFound desc = could not find container \"efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73\": container with ID starting with efe258b5213f155d9df2263750933c42fab3057538cb95cf6b2ae02bdb134a73 not found: ID does not exist"
Oct 30 19:37:56.486258 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:56.486238    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c83bbf5b66/isvc-pmml-runtime-predictor-654bc497b5-q68mr"]
Oct 30 19:37:56.724913 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:37:56.724873    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:37:57.541871 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:37:57.541835    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" path="/var/lib/kubelet/pods/c3cf9a32-67f7-4484-a023-ee52c4050113/volumes"
Oct 30 19:38:05.304494 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:05.304447    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:38:15.304974 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:15.304937    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:38:25.304693 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:25.304652    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:38:26.726475 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:38:26.726432    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:38:27.537206 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:27.537160    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:38:37.537329 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:37.537265    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:38:47.548405 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:47.548360    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:38:55.526866 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:55.526825    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:38:55.527356 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:55.527121    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" containerID="cri-o://7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385" gracePeriod=30
Oct 30 19:38:55.527356 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:55.527147    4942 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kube-rbac-proxy" containerID="cri-o://c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d" gracePeriod=30
Oct 30 19:38:55.725409 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:55.725366    4942 generic.go:356] "Generic (PLEG): container finished" podID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerID="c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d" exitCode=0
Oct 30 19:38:55.725612 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:55.725429    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerDied","Data":"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"}
Oct 30 19:38:56.727458 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:38:56.727420    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:38:57.538208 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:38:57.538149    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 19:39:00.299117 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.299062    4942 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.87:8643/healthz\": dial tcp 10.132.0.87:8643: connect: connection refused"
Oct 30 19:39:00.486282 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.486244    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:39:00.554499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554379    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\") pod \"94927a25-4147-41fe-b0f9-b803bc54e6e9\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") "
Oct 30 19:39:00.554499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554447    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\" (UniqueName: \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\") pod \"94927a25-4147-41fe-b0f9-b803bc54e6e9\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") "
Oct 30 19:39:00.554499 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554479    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\") pod \"94927a25-4147-41fe-b0f9-b803bc54e6e9\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") "
Oct 30 19:39:00.554814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554546    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\") pod \"94927a25-4147-41fe-b0f9-b803bc54e6e9\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") "
Oct 30 19:39:00.554814 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554644    4942 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\") pod \"94927a25-4147-41fe-b0f9-b803bc54e6e9\" (UID: \"94927a25-4147-41fe-b0f9-b803bc54e6e9\") "
Oct 30 19:39:00.555054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.554996    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location" pod "94927a25-4147-41fe-b0f9-b803bc54e6e9" (UID: "94927a25-4147-41fe-b0f9-b803bc54e6e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:00.555054 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.555035    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert" pod "94927a25-4147-41fe-b0f9-b803bc54e6e9" (UID: "94927a25-4147-41fe-b0f9-b803bc54e6e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:00.555279 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.555081    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config" pod "94927a25-4147-41fe-b0f9-b803bc54e6e9" (UID: "94927a25-4147-41fe-b0f9-b803bc54e6e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:00.556978 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.556938    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls" pod "94927a25-4147-41fe-b0f9-b803bc54e6e9" (UID: "94927a25-4147-41fe-b0f9-b803bc54e6e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:00.557152 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.556968    4942 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68" pod "94927a25-4147-41fe-b0f9-b803bc54e6e9" (UID: "94927a25-4147-41fe-b0f9-b803bc54e6e9"). InnerVolumeSpecName "kube-api-access-z8z68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:00.655650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.655583    4942 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-cabundle-cert\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:00.655650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.655645    4942 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94927a25-4147-41fe-b0f9-b803bc54e6e9-kserve-provision-location\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:00.655650 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.655662    4942 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94927a25-4147-41fe-b0f9-b803bc54e6e9-proxy-tls\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:00.655917 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.655676    4942 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z68\" (UniqueName: \"kubernetes.io/projected/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-api-access-z8z68\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:00.655917 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.655689    4942 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94927a25-4147-41fe-b0f9-b803bc54e6e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-34.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:00.749805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.749759    4942 generic.go:356] "Generic (PLEG): container finished" podID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerID="7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385" exitCode=0
Oct 30 19:39:00.750043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.749831    4942 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"
Oct 30 19:39:00.750043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.749839    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerDied","Data":"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"}
Oct 30 19:39:00.750043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.749900    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj" event={"ID":"94927a25-4147-41fe-b0f9-b803bc54e6e9","Type":"ContainerDied","Data":"8c9278ac582b9fa5b03042f93662da97969b3130752ccbe0dc88d6dddc8c61cf"}
Oct 30 19:39:00.750043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.749929    4942 scope.go:122] "RemoveContainer" containerID="c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"
Oct 30 19:39:00.759629 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.759607    4942 scope.go:122] "RemoveContainer" containerID="7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"
Oct 30 19:39:00.769179 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.769154    4942 scope.go:122] "RemoveContainer" containerID="56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"
Oct 30 19:39:00.774941 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.774901    4942 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:39:00.779521 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.779477    4942 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-3deaac58e4/isvc-pmml-predictor-7fcfc6d686-7k9qj"]
Oct 30 19:39:00.780262 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.780237    4942 scope.go:122] "RemoveContainer" containerID="c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"
Oct 30 19:39:00.780643 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:39:00.780621    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d\": container with ID starting with c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d not found: ID does not exist" containerID="c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"
Oct 30 19:39:00.780687 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.780655    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d"} err="failed to get container status \"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d\": rpc error: code = NotFound desc = could not find container \"c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d\": container with ID starting with c27398bb07b0cafaa07a384bdd70f118f97cf89a0e80352a3608edd4115a9c9d not found: ID does not exist"
Oct 30 19:39:00.780687 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.780677    4942 scope.go:122] "RemoveContainer" containerID="7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"
Oct 30 19:39:00.781044 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:39:00.780985    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385\": container with ID starting with 7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385 not found: ID does not exist" containerID="7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"
Oct 30 19:39:00.781123 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.781041    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385"} err="failed to get container status \"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385\": rpc error: code = NotFound desc = could not find container \"7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385\": container with ID starting with 7d3adad34870a5dd1985dc5d7fb52ccb0ed4c65761bd7b666c5b9fd83a4fd385 not found: ID does not exist"
Oct 30 19:39:00.781123 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.781060    4942 scope.go:122] "RemoveContainer" containerID="56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"
Oct 30 19:39:00.781368 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:39:00.781347    4942 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487\": container with ID starting with 56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487 not found: ID does not exist" containerID="56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"
Oct 30 19:39:00.781414 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:00.781373    4942 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487"} err="failed to get container status \"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487\": rpc error: code = NotFound desc = could not find container \"56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487\": container with ID starting with 56bc3a873728f61b4f45637a5609e6c2a15e8c3cec0fc0e4b5d66f0fc1a52487 not found: ID does not exist"
Oct 30 19:39:01.542880 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:01.542836    4942 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" path="/var/lib/kubelet/pods/94927a25-4147-41fe-b0f9-b803bc54e6e9/volumes"
Oct 30 19:39:26.728105 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:39:26.728059    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:39:34.171721 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.171684    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zdcz8/must-gather-m4ws9"]
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172001    4942 cpu_manager.go:414] "containerMap: removing container" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="storage-initializer"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172028    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="storage-initializer"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172044    4942 cpu_manager.go:414] "containerMap: removing container" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kube-rbac-proxy"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172049    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kube-rbac-proxy"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172058    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="storage-initializer"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172063    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="storage-initializer"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172081    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172087    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172094    4942 cpu_manager.go:414] "containerMap: removing container" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172099    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172110    4942 cpu_manager.go:414] "containerMap: removing container" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kube-rbac-proxy"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172115    4942 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kube-rbac-proxy"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172186    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172194    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kserve-container"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172201    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cf9a32-67f7-4484-a023-ee52c4050113" containerName="kube-rbac-proxy"
Oct 30 19:39:34.172223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.172210    4942 memory_manager.go:365] "RemoveStaleState removing state" podUID="94927a25-4147-41fe-b0f9-b803bc54e6e9" containerName="kube-rbac-proxy"
Oct 30 19:39:34.175306 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.175289    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.183204 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.183177    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zdcz8\"/\"openshift-service-ca.crt\""
Oct 30 19:39:34.183756 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.183731    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zdcz8/must-gather-m4ws9"]
Oct 30 19:39:34.183991 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.183973    4942 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zdcz8\"/\"kube-root-ca.crt\""
Oct 30 19:39:34.184093 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.183994    4942 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zdcz8\"/\"default-dockercfg-rst5l\""
Oct 30 19:39:34.349312 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.349259    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-must-gather-output\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.349528 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.349433    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4vm\" (UniqueName: \"kubernetes.io/projected/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-kube-api-access-6c4vm\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.450839 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.450801    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4vm\" (UniqueName: \"kubernetes.io/projected/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-kube-api-access-6c4vm\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.451033 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.450845    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-must-gather-output\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.451223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.451203    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-must-gather-output\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.464351 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.464321    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4vm\" (UniqueName: \"kubernetes.io/projected/924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e-kube-api-access-6c4vm\") pod \"must-gather-m4ws9\" (UID: \"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e\") " pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.498574 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.498528    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zdcz8/must-gather-m4ws9"
Oct 30 19:39:34.780226 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.780075    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zdcz8/must-gather-m4ws9"]
Oct 30 19:39:34.785399 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:39:34.785361    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod924e58d5_b75a_4d2d_8aa6_7ba55cf3ad6e.slice/crio-b89cb4f83d3ae0a24d9c6b884c2443990811c4b38a729684b8cbc254b4f2de1a WatchSource:0}: Error finding container b89cb4f83d3ae0a24d9c6b884c2443990811c4b38a729684b8cbc254b4f2de1a: Status 404 returned error can't find the container with id b89cb4f83d3ae0a24d9c6b884c2443990811c4b38a729684b8cbc254b4f2de1a
Oct 30 19:39:34.898245 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:34.898207    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zdcz8/must-gather-m4ws9" event={"ID":"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e","Type":"ContainerStarted","Data":"b89cb4f83d3ae0a24d9c6b884c2443990811c4b38a729684b8cbc254b4f2de1a"}
Oct 30 19:39:35.912640 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:35.912588    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zdcz8/must-gather-m4ws9" event={"ID":"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e","Type":"ContainerStarted","Data":"5060482121cc73d33e4e8311f732d7181c1ff40c1a46df1b7bdaeba6b9df2025"}
Oct 30 19:39:35.913141 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:35.912656    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zdcz8/must-gather-m4ws9" event={"ID":"924e58d5-b75a-4d2d-8aa6-7ba55cf3ad6e","Type":"ContainerStarted","Data":"77a7039c2df3b7aa653f30a371c59d2eeb7b1d9fb050e4effa5723339adc23b9"}
Oct 30 19:39:35.932694 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:35.932641    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zdcz8/must-gather-m4ws9" podStartSLOduration=1.179367375 podStartE2EDuration="1.932626177s" totalImagesPullingTime="753.258802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:35.932201484 +0000 UTC m=+3762.510297177" watchObservedRunningTime="2026-10-30 19:39:35.932626177 +0000 UTC m=+3762.510721869"
Oct 30 19:39:37.728065 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:37.728031    4942 ???:1] "http2: server: error reading preface from client 10.0.134.34:50816: read tcp 10.0.134.34:10250->10.0.134.34:50816: read: connection reset by peer"
Oct 30 19:39:37.736588 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:37.736558    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qw889_0eb9d479-2705-4c31-84c1-503e24ae3a0e/global-pull-secret-syncer/0.log"
Oct 30 19:39:37.798168 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:37.798132    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-64w7x_e844bc94-4bbe-410b-9cb6-807b45c555df/connection-checker/0.log"
Oct 30 19:39:37.833000 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:37.832965    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7v9n7_2cb00238-6e40-41f7-b8c4-05f7ef60e913/connection-checker/0.log"
Oct 30 19:39:37.874638 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:37.874611    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j9q9f_28992bb6-3816-46b7-af81-3b1a3932432c/connection-checker/0.log"
Oct 30 19:39:38.190492 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:38.190449    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x5hb6_d3132166-1ec1-448e-98c2-b9844cc8ef00/konnectivity-agent/0.log"
Oct 30 19:39:38.271043 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:38.270993    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-34.us-east-2.compute.internal_76487939913b7836b4b3b35bf9597886/haproxy/0.log"
Oct 30 19:39:43.839451 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:43.839410    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-hn5jb_27812221-bc2b-4915-ae7d-105a48913d66/cluster-monitoring-operator/0.log"
Oct 30 19:39:44.214805 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:44.214774    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bfz8b_b8ad0765-0cf3-4882-9ced-c0f6280f3ca9/node-exporter/0.log"
Oct 30 19:39:44.261081 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:44.261001    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bfz8b_b8ad0765-0cf3-4882-9ced-c0f6280f3ca9/kube-rbac-proxy/0.log"
Oct 30 19:39:44.297149 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:44.297116    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bfz8b_b8ad0765-0cf3-4882-9ced-c0f6280f3ca9/init-textfile/0.log"
Oct 30 19:39:45.898300 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:45.898256    4942 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"]
Oct 30 19:39:45.904129 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:45.904102    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:45.914722 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:45.914680    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"]
Oct 30 19:39:46.045564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.045526    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-podres\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.045564 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.045571    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-lib-modules\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.045851 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.045590    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmbj\" (UniqueName: \"kubernetes.io/projected/9a555611-8409-4934-9308-339b28fb22d9-kube-api-access-qqmbj\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.045851 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.045733    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-sys\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.045851 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.045806    4942 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-proc\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146520 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146480    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-sys\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146541    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-proc\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146568    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-podres\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146601    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-lib-modules\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146611    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-sys\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146623    4942 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmbj\" (UniqueName: \"kubernetes.io/projected/9a555611-8409-4934-9308-339b28fb22d9-kube-api-access-qqmbj\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146637    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-podres\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146637    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-proc\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.146712 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.146702    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a555611-8409-4934-9308-339b28fb22d9-lib-modules\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.160126 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.160056    4942 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmbj\" (UniqueName: \"kubernetes.io/projected/9a555611-8409-4934-9308-339b28fb22d9-kube-api-access-qqmbj\") pod \"perf-node-gather-daemonset-lg4l6\" (UID: \"9a555611-8409-4934-9308-339b28fb22d9\") " pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.226148 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.226109    4942 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:46.344066 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.344001    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-8pmcl_606559f4-9044-4445-90ee-d29a202d2b43/networking-console-plugin/0.log"
Oct 30 19:39:46.522509 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.522471    4942 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"]
Oct 30 19:39:46.528067 ip-10-0-134-34 kubenswrapper[4942]: W1030 19:39:46.528003    4942 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a555611_8409_4934_9308_339b28fb22d9.slice/crio-ac4970a4356940f1376124c4c49dc1e51612d05f02fc1be52685a6902c6d9496 WatchSource:0}: Error finding container ac4970a4356940f1376124c4c49dc1e51612d05f02fc1be52685a6902c6d9496: Status 404 returned error can't find the container with id ac4970a4356940f1376124c4c49dc1e51612d05f02fc1be52685a6902c6d9496
Oct 30 19:39:46.977506 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.977469    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/2.log"
Oct 30 19:39:46.982374 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:46.982330    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-spj7q_5c6185b5-cdeb-49ae-a8a6-95a9bac1dc03/console-operator/3.log"
Oct 30 19:39:47.008862 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:47.008822    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6" event={"ID":"9a555611-8409-4934-9308-339b28fb22d9","Type":"ContainerStarted","Data":"0f4c221e7698b7062b463dcc3c0cd13333498acaff3c9cbf4f7fd5748c8167e6"}
Oct 30 19:39:47.009047 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:47.008888    4942 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6" event={"ID":"9a555611-8409-4934-9308-339b28fb22d9","Type":"ContainerStarted","Data":"ac4970a4356940f1376124c4c49dc1e51612d05f02fc1be52685a6902c6d9496"}
Oct 30 19:39:47.009106 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:47.009055    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:47.031778 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:47.031702    4942 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6" podStartSLOduration=2.031681095 podStartE2EDuration="2.031681095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:47.028859433 +0000 UTC m=+3773.606955126" watchObservedRunningTime="2026-10-30 19:39:47.031681095 +0000 UTC m=+3773.609776787"
Oct 30 19:39:48.168112 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:48.167981    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-qc5dv_200dacf8-0b68-43ea-87ec-e5f67becf695/volume-data-source-validator/0.log"
Oct 30 19:39:49.359620 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:49.359579    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc44q_bbbfe361-0dd8-4e1e-bf8f-cd48924ec904/dns/0.log"
Oct 30 19:39:49.392069 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:49.392035    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc44q_bbbfe361-0dd8-4e1e-bf8f-cd48924ec904/kube-rbac-proxy/0.log"
Oct 30 19:39:49.577049 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:49.577000    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzfd_6914d9ae-4112-4a25-9f37-aa04aa12facc/dns-node-resolver/0.log"
Oct 30 19:39:50.168321 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:50.168296    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96jgv_f00a8cdd-93b4-459e-b271-03c501a4c5e5/node-ca/0.log"
Oct 30 19:39:51.116279 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:51.116245    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77bc686d65-bdnkc_363e48de-1ba5-4934-a2e7-e38cae066bb8/router/0.log"
Oct 30 19:39:51.524951 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:51.524917    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cdsk_a09bbce7-7e03-435e-992a-c1d89acdbe5a/serve-healthcheck-canary/0.log"
Oct 30 19:39:52.041664 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:52.041628    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fs9gh_8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/insights-operator/0.log"
Oct 30 19:39:52.043787 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:52.043764    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fs9gh_8d1e8569-d080-44ba-8c4b-b5ae19eaa04c/insights-operator/1.log"
Oct 30 19:39:52.070579 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:52.070554    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4x55m_44aaf253-a06d-4d3d-aac8-c1a6424dd260/kube-rbac-proxy/0.log"
Oct 30 19:39:52.102497 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:52.102464    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4x55m_44aaf253-a06d-4d3d-aac8-c1a6424dd260/exporter/0.log"
Oct 30 19:39:52.135148 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:52.135122    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4x55m_44aaf253-a06d-4d3d-aac8-c1a6424dd260/extractor/0.log"
Oct 30 19:39:53.023516 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:53.023485    4942 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zdcz8/perf-node-gather-daemonset-lg4l6"
Oct 30 19:39:55.526897 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:55.526863    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-56ddc4c448-85lmc_7e72c297-077d-4ddd-9c97-895a84666e34/server/0.log"
Oct 30 19:39:56.304223 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:56.304182    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7f9f8cdc65-wzhdb_adbf527c-4b4a-4ffe-b22f-97ecd5c320b8/manager/0.log"
Oct 30 19:39:56.590470 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:56.590390    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-78bb9c6ff8-hz5fg_9af38e72-3b1d-4b7d-b581-0565244a8706/seaweedfs/0.log"
Oct 30 19:39:56.626188 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:56.626154    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-685c9f986c-qpf7p_80115876-a6e1-4f39-a6c9-6847373e70f9/seaweedfs-tls-custom/0.log"
Oct 30 19:39:56.676567 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:39:56.676525    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-5c74974785-7j28b_e2002c0d-d6bf-470e-befb-53a919063a18/seaweedfs-tls-serving/0.log"
Oct 30 19:39:56.730278 ip-10-0-134-34 kubenswrapper[4942]: E1030 19:39:56.728553    4942 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4x55m" containerName="extractor"
Oct 30 19:40:01.217198 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:40:01.217163    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8x557_e91a9ddf-06c5-45fa-8d2f-a565fa2f52fb/migrator/0.log"
Oct 30 19:40:01.653880 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:40:01.653772    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-79v5d_a9506e6a-0145-4e49-b3f1-acd79fc488f5/kube-storage-version-migrator-operator/1.log"
Oct 30 19:40:01.655895 ip-10-0-134-34 kubenswrapper[4942]: I1030 19:40:01.655872    4942 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-79v5d_a9506e6a-0145-4e49-b3f1-acd79fc488f5/kube-storage-version-migrator-operator/0.log"