Oct 31 00:28:16.609978 ip-10-0-135-193 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:28:17.042218 ip-10-0-135-193 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:17.042218 ip-10-0-135-193 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:28:17.042218 ip-10-0-135-193 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:17.042218 ip-10-0-135-193 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:17.046632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046508    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.046632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046632    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046637    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046641    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046644    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046646    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046649    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046652    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046655    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046657    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046660    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046662    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046665    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046668    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046670    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046690    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046693    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046696    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046699    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046702    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046705    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.046717 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046707    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046710    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046712    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046715    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046718    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046721    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046731    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046735    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046737    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046740    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046743    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046745    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046748    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046750    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046761    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046764    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046766    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046769    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046771    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046774    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.047176 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046776    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046779    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046781    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046784    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046786    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046791    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046795    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046798    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046800    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046802    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046805    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046808    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046810    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046813    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046816    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046818    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046821    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046824    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046826    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.047698 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046828    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046831    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046834    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046836    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046839    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046841    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046845    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046848    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046850    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046852    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046855    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046857    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046860    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046863    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046866    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046868    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046870    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046873    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046876    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046878    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.048146 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046880    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046883    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046885    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046889    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046894    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046897    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046901    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046905    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046909    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046912    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046915    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046918    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046920    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046923    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046926    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046928    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046931    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046933    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046936    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046938    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.048748 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046942    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046944    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046947    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046950    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046952    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.046954    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047722    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047729    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047733    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047735    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047738    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047741    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047744    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047747    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047750    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047752    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047755    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047757    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.049216 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047760    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047763    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047765    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047769    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047772    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047774    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047777    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047780    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047782    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047785    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047787    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047790    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047792    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047794    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047797    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047799    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047802    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047804    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047806    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047809    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.049644 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047811    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047814    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047816    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047819    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047822    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047824    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047826    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047829    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047831    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047834    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047836    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047838    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047841    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047844    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047847    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047850    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047852    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047855    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047857    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047860    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.050152 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047862    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047865    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047867    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047869    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047872    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047874    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047877    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047879    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047882    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047884    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047886    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047889    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047891    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047894    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047898    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047900    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047903    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047905    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047908    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047910    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.050632 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047913    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047916    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047918    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047921    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047923    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047926    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047928    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047931    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047934    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047937    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047940    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047942    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047944    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047947    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047949    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047952    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047954    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047956    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047959    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.051130 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047961    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047964    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047966    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047969    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047971    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047973    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047976    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047981    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047985    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047988    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047990    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047994    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047997    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.047999    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.048002    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049326    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049333    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049340    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049343    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.051601 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049346    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049349    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049352    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049356    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049358    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049361    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049364    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049366    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049369    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049371    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049374    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049377    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049380    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049382    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049385    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049387    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049390    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049392    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049395    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.052068 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049398    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049402    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049406    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049409    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049412    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049414    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049417    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049419    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049422    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049424    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049427    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049430    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049433    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049435    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049438    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049440    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049442    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049445    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049448    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049450    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.052519 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049453    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049455    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049457    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049460    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049462    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049465    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049467    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049469    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049472    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049474    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049476    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049479    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049482    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049484    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049486    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049489    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049492    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049494    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049497    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049499    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.053049 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049501    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049504    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049506    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049509    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049511    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049514    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049516    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049518    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049521    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049523    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049526    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049529    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049531    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049534    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049536    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049539    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049541    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049544    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049546    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.053515 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049548    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049551    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049553    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049556    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049558    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049561    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049563    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049566    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049568    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049570    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049573    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049575    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049577    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049580    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049582    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049584    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049587    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049589    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049594    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.053975 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049598    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049601    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049603    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049606    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.049609    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050661    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050688    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050695    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050700    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050705    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050708    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050713    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050717    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050720    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050723    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050727    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050731    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050734    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050737    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050740    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050744    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050747    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050750    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050755    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:28:17.054459 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050757    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050761    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050764    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050767    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050772    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050775    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050778    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050781    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050784    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050787    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050790    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050793    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050796    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050801    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050805    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050807    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050810    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050814    2582 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050817    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050822    2582 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050825    2582 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050828    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050831    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050834    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:28:17.055034 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050838    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050841    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050844    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050847    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050850    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050853    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050856    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050859    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050862    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050865    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050868    2582 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050871    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050874    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050878    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050881    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050884    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050887    2582 flags.go:64] FLAG: --help="false"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050889    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050892    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050896    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050898    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050902    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050905    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:28:17.055620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050908    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050911    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050914    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050917    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050920    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050923    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050926    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050929    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050931    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050934    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050937    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050940    2582 flags.go:64] FLAG: --lock-file=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050942    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050945    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050948    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050954    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050956    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050959    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050962    2582 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050965    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050968    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050971    2582 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050974    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050978    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050981    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:28:17.056180 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050985    2582 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050988    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050991    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050994    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.050997    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051000    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051003    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051005    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051013    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051016    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051019    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051022    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051024    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051027    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051030    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051033    2582 flags.go:64] FLAG: --port="10250"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051036    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051039    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0423b6e8d51d21c41"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051042    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051045    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051048    2582 flags.go:64] FLAG: --register-node="true"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051051    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051055    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051057    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051060    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051063    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:28:17.056771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051067    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051070    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051073    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051076    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051078    2582 flags.go:64] FLAG: --runonce="false"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051081    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051084    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051087    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051090    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051092    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051095    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051098    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051102    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051105    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051107    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051110    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051113    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051116    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051119    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051122    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051127    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051130    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051133    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051137    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051140    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:28:17.057376 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051143    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051146    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051148    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051151    2582 flags.go:64] FLAG: --v="2"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051156    2582 flags.go:64] FLAG: --version="false"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051160    2582 flags.go:64] FLAG: --vmodule=""
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051164    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051167    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051308    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051312    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051315    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051318    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051320    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051323    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051325    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051328    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051330    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051333    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051336    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051338    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051340    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.058010 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051343    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051345    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051348    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051351    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051354    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051356    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051361    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051364    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051367    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051369    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051372    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051375    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051377    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051380    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051382    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051385    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051387    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051390    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051393    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051397    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.058607 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051400    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051402    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051405    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051409    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051413    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051416    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051418    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051421    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051424    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051426    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051428    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051431    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051434    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051436    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051439    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051442    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051444    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051447    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051452    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051455    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.059390 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051457    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051460    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051462    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051466    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051469    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051472    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051475    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051477    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051480    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051482    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051485    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051488    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051491    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051494    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051496    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051499    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051501    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051504    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051506    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.059898 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051509    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051511    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051514    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051517    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051519    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051522    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051524    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051527    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051529    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051532    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051534    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051538    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051541    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051543    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051546    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051548    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051551    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051553    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051556    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051559    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.060355 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051561    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051563    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051566    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051568    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051572    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051574    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051576    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051579    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051582    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051584    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051587    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051589    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051591    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051594    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.051620    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051656    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.060855 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051660    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051664    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051666    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051669    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051671    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051689    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051692    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051695    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051699    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051702    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051704    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051707    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051709    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051712    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051716    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051719    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051722    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051725    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051728    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.061236 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051731    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051735    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051737    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051740    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051742    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051745    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051747    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051750    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051752    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051755    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051758    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051760    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051762    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051765    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051768    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051771    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051775    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051777    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051780    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051782    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.061712 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051785    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051789    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051792    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051795    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051797    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051799    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051803    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051805    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051808    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051810    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051812    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051815    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051818    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051822    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051825    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051827    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051830    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051832    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051835    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.062194 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051837    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051840    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051842    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051844    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051847    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051849    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051852    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051854    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051857    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051859    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051862    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051864    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051867    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051869    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051874    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051876    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051879    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051881    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051883    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051886    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.062637 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051888    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051890    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051893    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051895    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051898    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051900    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051904    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051906    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051909    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051911    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051913    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051916    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051918    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051921    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051923    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051926    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051928    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051931    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051933    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.063135 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051936    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051938    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051940    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051944    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051946    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051949    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051951    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.051955    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.060489    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.060504    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.060521    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.060526    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060583    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060588    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060591    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060594    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060597    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060599    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060602    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060606    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.063626 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060609    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060611    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060613    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060616    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060619    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060621    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060625    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060629    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060632    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060636    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060638    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060641    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060644    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060646    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060649    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060652    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060654    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060657    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060659    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.064105 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060662    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060665    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060667    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060670    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060687    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060690    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060693    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060695    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060698    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060700    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060702    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060705    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060707    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060710    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060712    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060715    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060717    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060720    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060722    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060724    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.064555 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060727    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060729    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060732    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060735    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060737    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060741    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060745    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060748    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060750    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060753    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060755    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060758    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060761    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060763    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060766    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060769    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060771    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060773    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060776    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.065137 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060778    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060781    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060783    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060786    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060788    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060791    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060793    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060796    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060798    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060800    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060803    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060805    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060807    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060810    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060813    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060815    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060818    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060820    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060823    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060825    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.065581 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060827    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060830    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060832    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060835    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060837    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060840    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060842    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060845    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060847    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060850    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060852    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060855    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060857    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060860    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060863    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060865    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060868    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060871    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060873    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.066065 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.060876    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.060904    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061045    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061049    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061052    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061055    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061057    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061062    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061065    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061067    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061070    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061072    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061075    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061077    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061080    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061082    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.066538 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061085    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061087    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061090    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061092    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061095    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061097    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061100    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061102    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061105    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061107    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061110    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061112    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061115    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061117    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061120    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061122    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061125    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061127    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061131    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.066933 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061136    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061139    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061142    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061145    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061147    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061154    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061156    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061159    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061162    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061164    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061167    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061169    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061172    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061174    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061177    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061179    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061182    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061184    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061187    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.067379 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061189    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061192    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061194    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061197    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061199    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061202    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061204    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061206    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061209    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061211    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061214    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061217    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061219    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061221    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061224    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061226    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061229    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061231    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061234    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061241    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.067852 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061244    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061247    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061250    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061252    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061255    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061257    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061259    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061262    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061264    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061267    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061269    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061271    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061275    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061279    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061282    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061284    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061287    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061289    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061291    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.068330 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061294    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061297    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061299    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061301    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061304    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061306    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061309    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061311    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061313    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061316    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061320    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061323    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061326    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061329    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061331    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.061354    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:17.068790 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061385    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061389    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061391    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061394    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061396    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061399    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061402    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061404    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061407    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061409    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061412    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061414    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061416    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061419    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061421    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061425    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061428    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061432    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061435    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:17.069166 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061438    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061440    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061443    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061445    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061448    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061450    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061453    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061455    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061459    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061462    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061464    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061467    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061470    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061472    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061475    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061477    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061480    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061482    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061485    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:17.069599 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061487    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061490    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061493    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061495    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061497    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061500    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061502    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061505    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061508    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061512    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061514    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061517    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061519    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061522    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061524    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061527    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061529    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061532    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061534    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061537    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:17.070104 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061539    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061543    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061545    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061548    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061551    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061554    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061556    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061559    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061561    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061567    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061570    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061572    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061575    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061578    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061582    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061586    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061589    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061592    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061595    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061597    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:17.070587 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061600    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061604    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061606    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061611    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061614    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061616    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061619    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061621    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061624    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061626    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061629    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061631    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061635    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061637    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061640    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061643    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061645    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061648    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061650    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:17.071073 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061653    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061655    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061658    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061660    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061664    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061667    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061669    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:17.061671    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.061741    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.065011    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.066067    2582 server.go:1014] "Starting client certificate rotation"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.066166    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.066202    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.068349    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:17.071589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.069447    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:17.081836 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.081819    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:28:17.085574 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.085562    2582 log.go:25] "Validated CRI v1 image API"
Oct 31 00:28:17.086469 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.086443    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:28:17.090641 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.090620    2582 fs.go:135] Filesystem UUIDs: map[1fd60f1f-eff5-46ee-8004-3ee3bad04174:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bb45d447-1e84-48b3-b4a2-7d85c8f04e91:/dev/nvme0n1p3]
Oct 31 00:28:17.090740 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.090640    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:28:17.096633 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.096498    2582 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:17.094997673 +0000 UTC m=+0.377030683 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100777 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247ef87707fd58390dcbb11c7e99d9 SystemUUID:ec247ef8-7707-fd58-390d-cbb11c7e99d9 BootID:a4ca3629-3cf9-4ad5-84a8-7ee6bfe48d5e Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:45:af:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:45:af:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:85:61:3d:60:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:28:17.096633 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.096625    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:28:17.096822 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.096752    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:28:17.100744 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.100718    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:28:17.100899 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.100747    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-193.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:28:17.100995 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.100904    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:28:17.100995 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.100918    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:28:17.100995 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.100974    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:17.101865 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.101852    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:17.103098 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.103084    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:28:17.107539 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107513    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:28:17.107719 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107707    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:28:17.107835 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107826    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:28:17.107886 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107879    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:28:17.107915 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107892    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:28:17.107915 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107902    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:28:17.107915 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107910    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:28:17.108016 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.107919    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:28:17.108249 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.108235    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:17.110763 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.110739    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:28:17.116998 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.116983    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:17.117919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.117906    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:28:17.117958 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.117938    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:28:17.118113 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118102    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:28:17.118165 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118119    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:28:17.118165 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118125    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:28:17.118165 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118143    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:28:17.118165 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118153    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:28:17.118165 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118159    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:28:17.118869 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118857    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:28:17.118869 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118869    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:28:17.118958 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118890    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:28:17.118958 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118895    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:28:17.118958 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118904    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:28:17.118958 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.118912    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:28:17.119708 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.119698    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:28:17.119750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.119709    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:28:17.123932 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.123918    2582 server.go:1263] "Started kubelet"
Oct 31 00:28:17.124031 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.123969    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:28:17.124122 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.124050    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:28:17.124244 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.124230    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:28:17.124643 ip-10-0-135-193 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:28:17.127755 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.125749    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-193.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:28:17.127755 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.126443    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:28:17.127755 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.126551    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:28:17.130433 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.130415    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:28:17.131785 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.131754    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:28:17.131916 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.131896    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-193.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:28:17.132911 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.131734    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-193.us-east-2.compute.internal.18e375b50ef241ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-193.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-193.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:17.123877357 +0000 UTC m=+0.405910367,LastTimestamp:2026-10-31 00:28:17.123877357 +0000 UTC m=+0.405910367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-193.us-east-2.compute.internal,}"
Oct 31 00:28:17.133986 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.133974    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:17.134586 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.134569    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:28:17.134852 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.134833    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.135433 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135416    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:28:17.135433 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135415    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135439    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135465    2582 factory.go:55] Registering systemd factory
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135480    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135522    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135529    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135705    2582 factory.go:153] Registering CRI-O factory
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135719    2582 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135765    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135778    2582 factory.go:103] Registering Raw factory
Oct 31 00:28:17.136478 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.135788    2582 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:28:17.136965 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.136951    2582 manager.go:319] Starting recovery of all containers
Oct 31 00:28:17.141702 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.141657    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-193.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:28:17.145144 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.145120    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:28:17.147728 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.147707    2582 manager.go:324] Recovery completed
Oct 31 00:28:17.148166 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.148140    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khrgk"
Oct 31 00:28:17.151879 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.151769    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.153472 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.153458    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.153545 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.153486    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.153545 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.153500    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.154117 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154102    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:28:17.154117 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154113    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:28:17.154117 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154116    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.154266 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154128    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:28:17.154857 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154840    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.154921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154871    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.154921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.154882    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.155243 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.155162    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-193.us-east-2.compute.internal.18e375b510b5d7b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-193.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-193.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-193.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:17.153472439 +0000 UTC m=+0.435505456,LastTimestamp:2026-10-31 00:28:17.153472439 +0000 UTC m=+0.435505456,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-193.us-east-2.compute.internal,}"
Oct 31 00:28:17.156249 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.156238    2582 policy_none.go:50] "Start"
Oct 31 00:28:17.156291 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.156253    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:28:17.156291 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.156262    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:28:17.158146 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.158127    2582 policy_none.go:44] "Start"
Oct 31 00:28:17.161098 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.161077    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khrgk"
Oct 31 00:28:17.199418 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.196706    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.200884    2582 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.200919    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.200930    2582 server.go:87] "Starting device plugin registration server"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.201157    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.201169    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.201298    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.201356    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.201362    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.202323    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.202370    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.214954    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.216252    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.216272    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.216287    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.216316    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 00:28:17.223201 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.223024    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:17.301297 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.301237    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.302059 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.302044    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.302158 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.302070    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.302158 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.302080    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.302158 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.302103    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.311067 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.311052    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.311121 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.311074    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-193.us-east-2.compute.internal\": node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.317621 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.317595    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"]
Oct 31 00:28:17.317700 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.317652    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.318358 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.318343    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.318425 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.318370    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.318425 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.318383    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.320370 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.320358    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.320508 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.320493    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.320557 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.320524    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.321088 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321071    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.321174 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321101    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.321174 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321115    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.321174 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321135    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.321174 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321154    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.321174 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.321168    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.323006 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.322989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.323097 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.323019    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:17.323806 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.323788    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:17.323873 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.323827    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:17.323873 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.323843    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:17.329283 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.329268    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.353977 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.353960    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.356741 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.356725    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.429874 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.429854    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.437199 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.437174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.437416 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.437211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.437483 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.437447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.530198 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.530173    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.537545 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.537633 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.537633 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.537745 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537636    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.537745 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537643    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.537745 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.537639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.630952 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.630907    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.658429 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.658412    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.660501 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.660486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:17.731443 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.731419    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.831968 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.831933    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.932536 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:17.932475    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:17.960954 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.960925    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:17.968296 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:17.968281    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:18.035290 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.035272    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:18.046463 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.046434    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:18.048503 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.048490    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:18.061970 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.061955    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:18.066101 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.066087    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:28:18.066216 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.066200    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:28:18.066253 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.066241    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:28:18.066285 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.066246    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:28:18.108389 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.108375    2582 apiserver.go:52] "Watching apiserver"
Oct 31 00:28:18.125267 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.125247    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:28:18.125646 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.125625    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-7k48d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal","openshift-multus/multus-9bm4l","openshift-network-operator/iptables-alerter-nlqbt","openshift-ovn-kubernetes/ovnkube-node-rmgdp","kube-system/konnectivity-agent-42vpj","kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal","openshift-dns/node-resolver-ck7z7","openshift-image-registry/node-ca-jjgcf","openshift-multus/multus-additional-cni-plugins-nmcll","openshift-multus/network-metrics-daemon-lhnbd","openshift-network-diagnostics/network-check-target-c6kkm","kube-system/global-pull-secret-syncer-2pb9q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"]
Oct 31 00:28:18.128063 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.128045    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.130040 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.130021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.130114 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.130081    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:18.131990 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.131963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.132799 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132783    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.132799 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132788    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:28:18.132921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132826    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.132921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132823    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:28:18.132921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132830    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:28:18.133018 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.132975    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzwtm\""
Oct 31 00:28:18.133983 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.133969    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.134063 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.134046    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:18.136150 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.136134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.138132 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.138118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.140143 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.140128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.141234 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.141306 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141306 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141306 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141456 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141456 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141521 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141521 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141581 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141581 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141640 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141640 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141640 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75gr\" (UniqueName: \"kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.141750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.141919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6nb\" (UniqueName: \"kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.141919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.141919 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.141993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswmc\" (UniqueName: \"kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.142379 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.142803 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.142803 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.142900 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.142811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.144775 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.144755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.145049 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.145178 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.145264 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.145323 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.145323 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.145423 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.145423 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.145517 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.145452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdf44\" (UniqueName: \"kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.146952 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.146936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.149247 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.149086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.149247 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.149157    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:18.151047 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.151030    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:28:18.151368 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.151355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:18.151492 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.151478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:18.167091 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.167039    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:17 +0000 UTC" deadline="2028-06-05 13:15:26.880876922 +0000 UTC"
Oct 31 00:28:18.167091 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.167080    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14004h47m8.713799858s"
Oct 31 00:28:18.182531 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.182513    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tv55"
Oct 31 00:28:18.195752 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.195733    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tv55"
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233047    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233047    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233140    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233150    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233183    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233206    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233212    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:28:18.233302 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233308    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233451    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233597    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56d49\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233640    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233643    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233688    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233704    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233778    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9bm5h\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233817    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233868    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233877    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.233974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233939    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dnwdb\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.233983    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4hxn\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234026    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234058    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234062    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234080    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cnz4r\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234196    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8g45\""
Oct 31 00:28:18.234407 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234220    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r62gb\""
Oct 31 00:28:18.234605 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234510    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b2pkn\""
Oct 31 00:28:18.234749 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.234731    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:28:18.236247 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.236228    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:28:18.245937 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.245917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.245944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.245960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.245985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246027    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246061    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.246104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246105    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd56\" (UniqueName: \"kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246163    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246288    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246349    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246345    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246405    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246452    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.246558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246596    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246709    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246712    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.246739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247019    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75gr\" (UniqueName: \"kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6nb\" (UniqueName: \"kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.247334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswmc\" (UniqueName: \"kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdf44\" (UniqueName: \"kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247730    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248190 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247795    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.247892    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.248521    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.248638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:18.748601522 +0000 UTC m=+2.030634544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.248703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjrh\" (UniqueName: \"kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.248813    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.248865    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.248985 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.248870    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.249444 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249416    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.249510 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249457    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.249510 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.249806 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.249806 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249793    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.249932 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249900    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.249991 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.249925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250069 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250125 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250313 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.250313 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.250313 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.250313 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:18.250490 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.250490 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksbs\" (UniqueName: \"kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.250490 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250490 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtjg\" (UniqueName: \"kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250865    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250912    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.250946 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.250944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251027    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251069    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251137    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251191    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.251367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwngb\" (UniqueName: \"kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251427    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251605    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.251729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.252128 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251747    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.252128 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.252128 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.251823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.253756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.253734    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.254476 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.254103    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.254476 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.254131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.254476 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.254152    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.254476 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.254269    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.256559 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.256019    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.256559 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.256055    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.256559 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.256339    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.265414 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.264169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdf44\" (UniqueName: \"kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.265542 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.265521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswmc\" (UniqueName: \"kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.266023 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.265996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6nb\" (UniqueName: \"kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.266417 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.266401    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75gr\" (UniqueName: \"kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.324099 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.324069    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:18.352088 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtjg\" (UniqueName: \"kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352191    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352229 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352257    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwngb\" (UniqueName: \"kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd56\" (UniqueName: \"kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352421    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.352504 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352380    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352522    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.352615    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352756    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352735    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.352790    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:18.852771635 +0000 UTC m=+2.134804646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352804    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.353017 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjrh\" (UniqueName: \"kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.352990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353093    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksbs\" (UniqueName: \"kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353399    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.353756 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353408    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.354271 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353462    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.354271 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353528    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.354271 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.353645    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.354655 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.354637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.354881 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.354867    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.355390 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.355377    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.365451 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.365432    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd56\" (UniqueName: \"kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.366195 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.366174    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwngb\" (UniqueName: \"kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.366195 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.366185    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksbs\" (UniqueName: \"kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.366309 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.366198    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtjg\" (UniqueName: \"kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.367607 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.367591    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:18.367717 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.367611    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:18.367717 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.367623    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:18.367717 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.367702    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:18.867667267 +0000 UTC m=+2.149700288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:18.370022 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.369999    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjrh\" (UniqueName: \"kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.386614 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.386585    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9318895ca5d45d45cb102c90334bd92.slice/crio-0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8 WatchSource:0}: Error finding container 0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8: Status 404 returned error can't find the container with id 0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8
Oct 31 00:28:18.387110 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.387088    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7e31f843ae335a5983ffff48ea08a5.slice/crio-14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478 WatchSource:0}: Error finding container 14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478: Status 404 returned error can't find the container with id 14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478
Oct 31 00:28:18.392195 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.392181    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:28:18.483522 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.483464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:18.493275 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.493255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:18.493459 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.493441    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd8b314_8c3e_45a3_9395_bd88d51d1b08.slice/crio-9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08 WatchSource:0}: Error finding container 9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08: Status 404 returned error can't find the container with id 9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08
Oct 31 00:28:18.500927 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.500909    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ad6861_24a8_40ff_8a4e_ee332a213b54.slice/crio-ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39 WatchSource:0}: Error finding container ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39: Status 404 returned error can't find the container with id ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39
Oct 31 00:28:18.507434 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.507416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:18.514931 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.514910    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed308ea_c7aa_4772_ab2e_458e6ef7a33d.slice/crio-ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6 WatchSource:0}: Error finding container ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6: Status 404 returned error can't find the container with id ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6
Oct 31 00:28:18.527466 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.527447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:18.533292 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.533276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:18.535140 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.535117    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e79d059_a2d5_419e_bb3a_de74bb0082e5.slice/crio-8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24 WatchSource:0}: Error finding container 8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24: Status 404 returned error can't find the container with id 8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24
Oct 31 00:28:18.541589 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.541568    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f0fe6f_7a3d_4966_95be_31c2ee3941a6.slice/crio-0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77 WatchSource:0}: Error finding container 0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77: Status 404 returned error can't find the container with id 0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77
Oct 31 00:28:18.553994 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.553970    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:18.572938 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.572918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:18.579534 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.579515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:18.581827 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.581802    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa170b9_6ca8_45d3_b0a0_7394bf27fff0.slice/crio-b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a WatchSource:0}: Error finding container b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a: Status 404 returned error can't find the container with id b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a
Oct 31 00:28:18.585038 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.585021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:18.587773 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.587754    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e47f56_d809_4b6a_861b_98b7bd8241fb.slice/crio-ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2 WatchSource:0}: Error finding container ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2: Status 404 returned error can't find the container with id ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2
Oct 31 00:28:18.593753 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:18.593733    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61a812b_5c90_419b_bf1d_6c2cad75af52.slice/crio-8e3232364fb11818ec8c831ec603b5063a604ff43a042686372485ed6dfc1565 WatchSource:0}: Error finding container 8e3232364fb11818ec8c831ec603b5063a604ff43a042686372485ed6dfc1565: Status 404 returned error can't find the container with id 8e3232364fb11818ec8c831ec603b5063a604ff43a042686372485ed6dfc1565
Oct 31 00:28:18.756244 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.756188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:18.756342 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.756291    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:18.756342 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.756337    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:19.756324938 +0000 UTC m=+3.038357936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:18.857312 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.857287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:18.857446 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.857430    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:18.857497 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.857487    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:19.857471707 +0000 UTC m=+3.139504710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:18.958088 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:18.958046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:18.958233 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.958194    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:18.958233 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.958212    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:18.958233 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.958221    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:18.958328 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:18.958272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:19.958258083 +0000 UTC m=+3.240291084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:19.198955 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.198665    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:18 +0000 UTC" deadline="2028-04-14 06:20:17.917187725 +0000 UTC"
Oct 31 00:28:19.198955 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.198807    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12749h51m58.718386533s"
Oct 31 00:28:19.250465 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.250332    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp" event={"ID":"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0","Type":"ContainerStarted","Data":"b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a"}
Oct 31 00:28:19.288060 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.288017    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nlqbt" event={"ID":"bed308ea-c7aa-4772-ab2e-458e6ef7a33d","Type":"ContainerStarted","Data":"ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6"}
Oct 31 00:28:19.305500 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.305465    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bm4l" event={"ID":"c6ad6861-24a8-40ff-8a4e-ee332a213b54","Type":"ContainerStarted","Data":"ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39"}
Oct 31 00:28:19.316147 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.316082    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerStarted","Data":"9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08"}
Oct 31 00:28:19.317907 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.317876    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"2b7e31f843ae335a5983ffff48ea08a5","Type":"ContainerStarted","Data":"14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478"}
Oct 31 00:28:19.323638 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.323609    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7k48d" event={"ID":"c61a812b-5c90-419b-bf1d-6c2cad75af52","Type":"ContainerStarted","Data":"8e3232364fb11818ec8c831ec603b5063a604ff43a042686372485ed6dfc1565"}
Oct 31 00:28:19.343008 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.342973    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjgcf" event={"ID":"e5ba2243-f40b-4f95-a955-bd82c885d457","Type":"ContainerStarted","Data":"4b2c563768268d3226f232fc7dbff8d744160f0aa235d38418d788f17d995ffa"}
Oct 31 00:28:19.346771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.346744    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42vpj" event={"ID":"25f0fe6f-7a3d-4966-95be-31c2ee3941a6","Type":"ContainerStarted","Data":"0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77"}
Oct 31 00:28:19.350384 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.350345    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24"}
Oct 31 00:28:19.358557 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.358528    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerStarted","Data":"0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8"}
Oct 31 00:28:19.360713 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.360651    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ck7z7" event={"ID":"22e47f56-d809-4b6a-861b-98b7bd8241fb","Type":"ContainerStarted","Data":"ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2"}
Oct 31 00:28:19.522286 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.522209    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:19.765689 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.765636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:19.765933 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.765856    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:19.765933 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.765921    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:21.765902502 +0000 UTC m=+5.047935517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:19.868104 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.868015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:19.868267 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.868227    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:19.868377 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.868349    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:21.86830946 +0000 UTC m=+5.150342474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:19.969018 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:19.968979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:19.969207 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.969172    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:19.969207 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.969193    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:19.969207 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.969207    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:19.969466 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:19.969266    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:21.969248672 +0000 UTC m=+5.251281676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:20.199431 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.199336    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:18 +0000 UTC" deadline="2028-05-23 08:13:55.092678889 +0000 UTC"
Oct 31 00:28:20.199431 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.199382    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13687h45m34.893302314s"
Oct 31 00:28:20.217173 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.217143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:20.217326 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:20.217306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:20.217814 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.217743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:20.217916 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:20.217883    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:20.217975 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.217956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:20.218052 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:20.218027    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:20.227450 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.227430    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:20.922881 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:20.922821    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:21.785691 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:21.785656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:21.786213 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.785838    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:21.786213 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.785913    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:25.78589294 +0000 UTC m=+9.067925959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:21.886578 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:21.886538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:21.894991 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.887125    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:21.894991 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.887227    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:25.887193405 +0000 UTC m=+9.169226422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:21.987428 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:21.987345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:21.987584 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.987535    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:21.987584 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.987554    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:21.987584 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.987567    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:21.987788 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:21.987628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:25.987609411 +0000 UTC m=+9.269642411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:22.217525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:22.217545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:22.217566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:22.217698    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:22.217770    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:22.218037 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:22.217858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:23.404799 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:23.404584    2582 generic.go:356] "Generic (PLEG): container finished" podID="f9318895ca5d45d45cb102c90334bd92" containerID="8e6a1946ab4519ad5233c9072ed090c769e3867187471f0dbba786f4e6e79c68" exitCode=0
Oct 31 00:28:23.405188 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:23.404827    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerDied","Data":"8e6a1946ab4519ad5233c9072ed090c769e3867187471f0dbba786f4e6e79c68"}
Oct 31 00:28:23.411466 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:23.410841    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="92fdffe787acc874f8e4fce3afe3393b90a0806752bc5e485ab13a907813d720" exitCode=0
Oct 31 00:28:23.411466 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:23.410883    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"92fdffe787acc874f8e4fce3afe3393b90a0806752bc5e485ab13a907813d720"}
Oct 31 00:28:24.216611 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.216581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:24.216780 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:24.216739    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:24.217164 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.217142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:24.217288 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:24.217267    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:24.217346 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.217331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:24.217428 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:24.217408    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:24.418635 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.418596    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"2b7e31f843ae335a5983ffff48ea08a5","Type":"ContainerStarted","Data":"d0aad7d9dcdf0ef3f5d47f018fea9aa9d5f32a8d42d54627640c1419f40c7c81"}
Oct 31 00:28:24.422897 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.422518    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-42vpj" event={"ID":"25f0fe6f-7a3d-4966-95be-31c2ee3941a6","Type":"ContainerStarted","Data":"00c0c015b365ab6dfee39d1ac9f4197378557cc53bd3f0cde3a3a19443d38e85"}
Oct 31 00:28:24.427431 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.427025    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerStarted","Data":"1a348f524960f8d758687fbd20017ac281bfc5740d1f4dd365d8df07cabd2961"}
Oct 31 00:28:24.457888 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.456911    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal" podStartSLOduration=6.456893674 podStartE2EDuration="6.456893674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.437923796 +0000 UTC m=+7.719956817" watchObservedRunningTime="2026-10-31 00:28:24.456893674 +0000 UTC m=+7.738926697"
Oct 31 00:28:24.457888 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.457529    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" podStartSLOduration=6.457521667 podStartE2EDuration="6.457521667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.456733956 +0000 UTC m=+7.738766977" watchObservedRunningTime="2026-10-31 00:28:24.457521667 +0000 UTC m=+7.739554688"
Oct 31 00:28:24.484696 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:24.484625    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-42vpj" podStartSLOduration=2.836043025 podStartE2EDuration="7.484609069s" totalImagesPullingTime="4.648566044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.483473986 +0000 UTC m=+7.765507012" watchObservedRunningTime="2026-10-31 00:28:24.484609069 +0000 UTC m=+7.766642090"
Oct 31 00:28:25.438273 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:25.438235    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjgcf" event={"ID":"e5ba2243-f40b-4f95-a955-bd82c885d457","Type":"ContainerStarted","Data":"9e59fc5dce70d9a4298aa71af1ef32e5b11b5fa88c0a257b02d9e346c1b54b36"}
Oct 31 00:28:25.823565 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:25.823492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:25.823762 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:25.823597    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:25.823762 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:25.823643    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:33.823631581 +0000 UTC m=+17.105664579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:25.924646 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:25.924604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:25.924839 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:25.924766    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:25.924839 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:25.924833    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:33.924815164 +0000 UTC m=+17.206848165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:26.025628 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:26.025588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:26.025815 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.025792    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:26.025887 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.025818    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:26.025887 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.025831    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:26.025994 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.025889    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:34.025870524 +0000 UTC m=+17.307903526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:26.217098 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:26.217022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:26.217221 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.217156    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:26.217592 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:26.217569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:26.217723 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.217695    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:26.217798 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:26.217759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:26.217848 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:26.217824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:26.443451 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:26.443414    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp" event={"ID":"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0","Type":"ContainerStarted","Data":"0807ccfacd923d3890e5006af64ae40d1361f844282cc160af0fc68a00c5d253"}
Oct 31 00:28:27.151085 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.151046    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:28:27.206885 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.206788    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:27.151067067Z","UUID":"f911b3d1-c5b7-4393-b847-7e3502b9ab5b","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:28:27.208282 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.208259    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:28:27.208402 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.208290    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:28:27.447634 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.447598    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp" event={"ID":"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0","Type":"ContainerStarted","Data":"f2df0c6708a2322f499ca986d761910eca1f01369418e6f98c048937ede0e806"}
Oct 31 00:28:27.450494 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.450442    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bm4l" event={"ID":"c6ad6861-24a8-40ff-8a4e-ee332a213b54","Type":"ContainerStarted","Data":"cc0cf9eb7448285f1c9edb43d553086450d90389a3251d435fe577a5bfe1c9e3"}
Oct 31 00:28:27.473141 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.473084    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9bm4l" podStartSLOduration=2.256542585 podStartE2EDuration="10.473073331s" totalImagesPullingTime="8.216530746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:27.472327973 +0000 UTC m=+10.754360994" watchObservedRunningTime="2026-10-31 00:28:27.473073331 +0000 UTC m=+10.755106355"
Oct 31 00:28:27.473423 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:27.473399    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jjgcf" podStartSLOduration=4.569700676 podStartE2EDuration="10.473394256s" totalImagesPullingTime="5.90369358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:25.457096654 +0000 UTC m=+8.739129673" watchObservedRunningTime="2026-10-31 00:28:27.473394256 +0000 UTC m=+10.755427340"
Oct 31 00:28:28.217661 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.217423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:28.217835 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.217499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:28.217835 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:28.217742    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:28.217835 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:28.217768    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:28.217835 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.217510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:28.218034 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:28.217850    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:28.433037 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.432424    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:28.433202 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.433070    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:28.455084 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.455050    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ck7z7" event={"ID":"22e47f56-d809-4b6a-861b-98b7bd8241fb","Type":"ContainerStarted","Data":"bc640e85a89b48e01cf8948a6fbcac39b6a3e20b326c4431a4a5a34437d150d8"}
Oct 31 00:28:28.458668 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.458202    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp" event={"ID":"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0","Type":"ContainerStarted","Data":"7ee99a7cb2712d7a5d8f72c914c17c57044805228f61a860a83c53bb1f9767ee"}
Oct 31 00:28:28.460730 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.460702    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="58fcba4e12f3e6addbd27e0eb9973badfdd60d0ed833625021e5466304a49898" exitCode=0
Oct 31 00:28:28.460841 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.460787    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"58fcba4e12f3e6addbd27e0eb9973badfdd60d0ed833625021e5466304a49898"}
Oct 31 00:28:28.474367 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.474285    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ck7z7" podStartSLOduration=2.481331714 podStartE2EDuration="11.474269054s" totalImagesPullingTime="8.99293734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:28.47407559 +0000 UTC m=+11.756108611" watchObservedRunningTime="2026-10-31 00:28:28.474269054 +0000 UTC m=+11.756302075"
Oct 31 00:28:28.496431 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:28.496391    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp" podStartSLOduration=2.228597668 podStartE2EDuration="11.496376567s" totalImagesPullingTime="9.267778899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:28.495839058 +0000 UTC m=+11.777872080" watchObservedRunningTime="2026-10-31 00:28:28.496376567 +0000 UTC m=+11.778409602"
Oct 31 00:28:29.466089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:29.465666    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nlqbt" event={"ID":"bed308ea-c7aa-4772-ab2e-458e6ef7a33d","Type":"ContainerStarted","Data":"d0ae0252145c15ca882f109f054e986a85aafa1804b9bed0a7b48f209fb72f29"}
Oct 31 00:28:29.470008 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:29.469905    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7k48d" event={"ID":"c61a812b-5c90-419b-bf1d-6c2cad75af52","Type":"ContainerStarted","Data":"f8e20e014a6db9c73c0a25242c8665564878b74d07409abb8e29f553655a716b"}
Oct 31 00:28:29.509456 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:29.509408    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nlqbt" podStartSLOduration=3.689668325 podStartE2EDuration="12.509389699s" totalImagesPullingTime="8.819721374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:29.486549669 +0000 UTC m=+12.768582688" watchObservedRunningTime="2026-10-31 00:28:29.509389699 +0000 UTC m=+12.791422722"
Oct 31 00:28:30.196514 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.196470    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:30.196670 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.196625    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:30.197156 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.197142    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:30.216514 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.216485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:30.216644 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.216495    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7k48d" podStartSLOduration=2.442647747 podStartE2EDuration="13.216482377s" totalImagesPullingTime="10.77383463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:29.509322791 +0000 UTC m=+12.791355812" watchObservedRunningTime="2026-10-31 00:28:30.216482377 +0000 UTC m=+13.498515396"
Oct 31 00:28:30.216644 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.216543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:30.216644 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.216637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:30.216837 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:30.216654    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:30.216837 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:30.216767    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:30.216951 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:30.216858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:30.474408 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.474324    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="d1e0f36cefe71617c71f22b5b67823dc38d3e32f53fc80ee6c059266e5bff5f9" exitCode=0
Oct 31 00:28:30.474408 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:30.474387    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"d1e0f36cefe71617c71f22b5b67823dc38d3e32f53fc80ee6c059266e5bff5f9"}
Oct 31 00:28:32.217096 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:32.216910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:32.217574 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:32.216994    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:32.217574 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:32.217193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:32.217574 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:32.217004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:32.217574 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:32.217283    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:32.217574 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:32.217347    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:32.481393 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:32.481322    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="9f6b6ae756d29aae2b54e337a608d662fc1a70a92a2b4bb9738286fc75bfec72" exitCode=0
Oct 31 00:28:32.481393 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:32.481384    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"9f6b6ae756d29aae2b54e337a608d662fc1a70a92a2b4bb9738286fc75bfec72"}
Oct 31 00:28:33.885969 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:33.885936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:33.886366 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:33.886084    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:33.886366 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:33.886138    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret podName:cbfee234-053b-4291-aecd-5c197dded041 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.886123124 +0000 UTC m=+33.168156136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret") pod "global-pull-secret-syncer-2pb9q" (UID: "cbfee234-053b-4291-aecd-5c197dded041") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:33.987011 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:33.986951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:33.987121 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:33.987103    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:33.987188 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:33.987170    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.987151888 +0000 UTC m=+33.269184899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:34.087296 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.087272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:34.087418 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.087403    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:34.087487 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.087427    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:34.087487 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.087438    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h44ts for pod openshift-network-diagnostics/network-check-target-c6kkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:34.087592 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.087496    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts podName:3625a6a6-66db-49b1-8fb6-fae35972b557 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.087478941 +0000 UTC m=+33.369511943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h44ts" (UniqueName: "kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts") pod "network-check-target-c6kkm" (UID: "3625a6a6-66db-49b1-8fb6-fae35972b557") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:34.216981 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.216768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:34.217076 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.217029    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:34.217076 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.216821    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:34.217156 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.217105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:34.217156 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.216817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:34.217156 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:34.217152    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:34.489331 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:34.489658 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489636    2582 generic.go:356] "Generic (PLEG): container finished" podID="0e79d059-a2d5-419e-bb3a-de74bb0082e5" containerID="7bb4d212ab315d177657914c406cc154c82ab5e19686e70fa8403b85f46cb363" exitCode=1
Oct 31 00:28:34.489735 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489715    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"72b38edac35431dccfd006c1fbb76e74e1973e8e89757ccda50c813a5a516022"}
Oct 31 00:28:34.489840 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489823    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"dcfd4f9356437e517e1d7fc2a8de8c1633156c5c5936d09eefebd6b60aea390e"}
Oct 31 00:28:34.489947 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489860    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"324161e693af7424cbf7033b39842f43f9e0b02eb53443ed32c498caa19c99b0"}
Oct 31 00:28:34.489947 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489894    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"8d0c9008f45a22abca4d3084ee6f1b06f6b39c7e98644c575961e1271bbf03a8"}
Oct 31 00:28:34.489947 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489923    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerDied","Data":"7bb4d212ab315d177657914c406cc154c82ab5e19686e70fa8403b85f46cb363"}
Oct 31 00:28:34.490067 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.489959    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"3011ff293697db5325e0b62e633fb9a6b3ed00a8db0a2c64af0d0172156dd420"}
Oct 31 00:28:34.492299 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:34.492277    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerStarted","Data":"076728c21752f8de1cad538b2630d5cb23b084cc944af00963e8cc23b55d61c9"}
Oct 31 00:28:35.497493 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:35.497456    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="076728c21752f8de1cad538b2630d5cb23b084cc944af00963e8cc23b55d61c9" exitCode=0
Oct 31 00:28:35.497493 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:35.497479    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="b855ed38811b18a9ec2958a06124d893a3977a4410af9cad71acdcf7c9ae25df" exitCode=0
Oct 31 00:28:35.498225 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:35.497513    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"076728c21752f8de1cad538b2630d5cb23b084cc944af00963e8cc23b55d61c9"}
Oct 31 00:28:35.498225 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:35.497543    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerDied","Data":"b855ed38811b18a9ec2958a06124d893a3977a4410af9cad71acdcf7c9ae25df"}
Oct 31 00:28:36.217089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.217058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:36.217089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.217086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:36.217089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.217099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:36.217331 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:36.217175    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:36.217331 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:36.217238    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:36.217331 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:36.217312    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:36.503636 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.503605    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmcll" event={"ID":"9cd8b314-8c3e-45a3-9395-bd88d51d1b08","Type":"ContainerStarted","Data":"40d64fdd6447c503cce77f9e769cad610e5cf0c3adc3dc8808842c210f4d20fd"}
Oct 31 00:28:36.506605 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.506587    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:36.506896 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.506875    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"e75e2f54fbe332ac64574cae49a840f571c1ff1e233bbd895d55ce4b195882bf"}
Oct 31 00:28:36.533406 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:36.533356    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nmcll" podStartSLOduration=8.281787538 podStartE2EDuration="19.533339628s" totalImagesPullingTime="11.25155209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:36.5320938 +0000 UTC m=+19.814126819" watchObservedRunningTime="2026-10-31 00:28:36.533339628 +0000 UTC m=+19.815372648"
Oct 31 00:28:38.217272 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:38.217247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:38.217272 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:38.217269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:38.217785 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:38.217356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:38.217785 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:38.217363    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:38.217785 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:38.217419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:38.217785 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:38.217484    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:39.516296 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:39.516121    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:39.516940 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:39.516627    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"eac71c74d9584f6d3d7a9d835537e11538cd8005c5ef176e0ea3d2e2db960a7f"}
Oct 31 00:28:39.516940 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:39.516924    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:39.517058 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:39.516983    2582 scope.go:122] "RemoveContainer" containerID="7bb4d212ab315d177657914c406cc154c82ab5e19686e70fa8403b85f46cb363"
Oct 31 00:28:39.533758 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:39.533740    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:40.216843 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.216814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:40.216988 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.216814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:40.216988 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.216925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:40.216988 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.216936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:40.217150 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.217015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:40.217150 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.217110    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:40.523089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.523037    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:40.523505 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.523378    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" event={"ID":"0e79d059-a2d5-419e-bb3a-de74bb0082e5","Type":"ContainerStarted","Data":"6b7265c82c423042f0a39b39484279ba932abddb059695decef6aedd2eeddbf8"}
Oct 31 00:28:40.523505 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.523467    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:40.523723 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.523706    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:40.542717 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.542689    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:40.561452 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.561431    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2pb9q"]
Oct 31 00:28:40.561558 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.561522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:40.561624 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.561607    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:40.565163 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.565135    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhnbd"]
Oct 31 00:28:40.565269 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.565240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:40.565364 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.565339    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:40.565947 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.565924    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6kkm"]
Oct 31 00:28:40.566027 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.566016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:40.566119 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:40.566102    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:40.574153 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:40.574119    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp" podStartSLOduration=8.241838257 podStartE2EDuration="23.574108588s" totalImagesPullingTime="15.332270331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:40.573660315 +0000 UTC m=+23.855693346" watchObservedRunningTime="2026-10-31 00:28:40.574108588 +0000 UTC m=+23.856141605"
Oct 31 00:28:41.525994 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:41.525966    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:42.216783 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:42.216745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:42.216954 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:42.216845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:42.216954 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:42.216858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:42.216954 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:42.216857    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:42.216954 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:42.216921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:42.217103 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:42.217005    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:42.527849 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:42.527779    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:44.217429 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:44.217402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:44.217807 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:44.217402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:44.217807 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:44.217515    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:44.217807 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:44.217599    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:44.217807 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:44.217402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:44.217807 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:44.217703    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:45.343768 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:45.343578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:45.547997 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:45.547969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.217089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:46.217052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:46.217089 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:46.217081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:46.217334 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:46.217087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:46.217334 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:46.217175    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6kkm" podUID="3625a6a6-66db-49b1-8fb6-fae35972b557"
Oct 31 00:28:46.217334 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:46.217209    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhnbd" podUID="a862323a-20e5-41f7-86bf-79f2964d1ec5"
Oct 31 00:28:46.217334 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:46.217270    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2pb9q" podUID="cbfee234-053b-4291-aecd-5c197dded041"
Oct 31 00:28:46.686747 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:46.686722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.874132 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:46.874039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:47.043124 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.043092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:47.478610 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.478547    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeReady"
Oct 31 00:28:47.478750 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.478638    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:28:47.537965 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.537942    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jrkck"]
Oct 31 00:28:47.543382 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.543359    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-76d7d"]
Oct 31 00:28:47.543512 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.543499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.546482 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.546460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.553737 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.553697    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jrkck"]
Oct 31 00:28:47.554471 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554452    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:28:47.554559 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554500    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nchkl\""
Oct 31 00:28:47.554620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554602    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:28:47.554620 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554609    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:28:47.554738 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554642    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:28:47.554738 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554657    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:28:47.554825 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.554802    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8j7n8\""
Oct 31 00:28:47.558771 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.558714    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76d7d"]
Oct 31 00:28:47.692740 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.692713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.693171 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.692768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.693171 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.692793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.693171 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.692902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwsw\" (UniqueName: \"kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.693171 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.692932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47sq\" (UniqueName: \"kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.693171 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.693020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794051 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.793981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47sq\" (UniqueName: \"kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794051 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794210 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.794210 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794210 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794210 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwsw\" (UniqueName: \"kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.794399 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:47.794308    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:47.794399 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:47.794381    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:48.294362043 +0000 UTC m=+31.576395058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:47.794513 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.794790 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.794773    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.798398 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.798381    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.807488 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.807460    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwsw\" (UniqueName: \"kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:47.807589 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.807545    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47sq\" (UniqueName: \"kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:47.855586 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:47.855562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:48.144266 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.144200    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jrkck"]
Oct 31 00:28:48.147247 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:48.147225    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e796e1_0c4b_455c_906d_98ee4c141fe0.slice/crio-a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076 WatchSource:0}: Error finding container a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076: Status 404 returned error can't find the container with id a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076
Oct 31 00:28:48.216664 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.216640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:48.216806 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.216699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:48.216806 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.216707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:48.221877 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.221846    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:28:48.221989 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.221890    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:28:48.222052 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.222019    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:28:48.222403 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.222389    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qx27s\""
Oct 31 00:28:48.222566 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.222553    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hxhg5\""
Oct 31 00:28:48.222639 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.222622    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bqfb\""
Oct 31 00:28:48.222722 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.222627    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:28:48.297354 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.297332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:48.297457 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:48.297444    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:48.297503 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:48.297499    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.297486396 +0000 UTC m=+32.579519408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:48.367880 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.367860    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:48.368069 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.368044    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:48.382695 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.382665    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:48.546312 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:48.546286    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jrkck" event={"ID":"c3e796e1-0c4b-455c-906d-98ee4c141fe0","Type":"ContainerStarted","Data":"a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076"}
Oct 31 00:28:49.304434 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.304399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:49.304874 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:49.304520    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.304874 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:49.304595    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:51.304577203 +0000 UTC m=+34.586610205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.696060 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.696028    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:28:49.730662 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.729570    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnddf"]
Oct 31 00:28:49.730662 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.729732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.738244 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.738208    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.738365 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.738326    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:28:49.738365 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.738338    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.738511 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.738494    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rmr2b\""
Oct 31 00:28:49.740849 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.740828    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:28:49.741101 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.741087    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:28:49.741168 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.741113    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnddf"]
Oct 31 00:28:49.741212 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.741195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.745464 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.745422    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:28:49.745870 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.745852    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:28:49.746006 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.745990    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wfrwt\""
Oct 31 00:28:49.766853 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.766830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:49.809091 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.809065    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.809209 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.809101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.809209 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.809126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.809209 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.809145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.909732 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.909672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:49.909898 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.909749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.909898 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.909775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.909898 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.909813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.910038 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.909968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910038 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.910124 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910124 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910284 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910758 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910740    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910758 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910746    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.910915 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.910820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9rk\" (UniqueName: \"kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:49.911144 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.911118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:49.911363 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.911345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:49.912909 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.912887    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:49.926146 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:49.926100    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:50.011692 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011692 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9rk\" (UniqueName: \"kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011797    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.011835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.011862 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011852    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:50.012158 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011894    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.511876735 +0000 UTC m=+33.793909748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.012158 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011892    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:50.012158 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011921    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.511905723 +0000 UTC m=+33.793938738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:50.012158 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.011949    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs podName:a862323a-20e5-41f7-86bf-79f2964d1ec5 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:22.011932899 +0000 UTC m=+65.293965915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs") pod "network-metrics-daemon-lhnbd" (UID: "a862323a-20e5-41f7-86bf-79f2964d1ec5") : secret "metrics-daemon-secret" not found
Oct 31 00:28:50.012158 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.012064    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.025719 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.025670    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9rk\" (UniqueName: \"kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.054424 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.054399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:50.074729 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.074668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:50.112857 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.112787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:50.115831 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.115810    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:50.270577 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.270492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.354631 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.354475    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:28:50.357398 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:50.357368    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3e94f4_0e2f_4c2d_b9c1_f44a35ca4971.slice/crio-155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2 WatchSource:0}: Error finding container 155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2: Status 404 returned error can't find the container with id 155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2
Oct 31 00:28:50.404019 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.403995    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2pb9q"]
Oct 31 00:28:50.407034 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:50.407004    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfee234_053b_4291_aecd_5c197dded041.slice/crio-035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d WatchSource:0}: Error finding container 035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d: Status 404 returned error can't find the container with id 035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d
Oct 31 00:28:50.413764 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.413743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:50.427921 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.427895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.517129 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.517102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.517257 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.517159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:50.517257 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.517233    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:50.517370 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.517275    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.517370 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.517281    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:51.517266205 +0000 UTC m=+34.799299206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:50.517370 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:50.517353    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:51.517334141 +0000 UTC m=+34.799367143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.553753 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.553572    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2pb9q" event={"ID":"cbfee234-053b-4291-aecd-5c197dded041","Type":"ContainerStarted","Data":"035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d"}
Oct 31 00:28:50.554981 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.554950    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" event={"ID":"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971","Type":"ContainerStarted","Data":"155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2"}
Oct 31 00:28:50.616797 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.616766    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"]
Oct 31 00:28:50.628390 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.628336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.649885 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.649830    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"]
Oct 31 00:28:50.650024 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.649958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:50.658121 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.658081    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:50.658614 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.658593    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pmz5p\""
Oct 31 00:28:50.659073 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.659055    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:28:50.723628 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.723600    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6kkm"]
Oct 31 00:28:50.727106 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:50.727076    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3625a6a6_66db_49b1_8fb6_fae35972b557.slice/crio-651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5 WatchSource:0}: Error finding container 651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5: Status 404 returned error can't find the container with id 651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5
Oct 31 00:28:50.782373 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.782340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.820441 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.820364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd577\" (UniqueName: \"kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577\") pod \"migrator-66df5d496b-429vc\" (UID: \"f4c055f3-44ac-423d-9c0c-85383b5d8d94\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:50.855136 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.855111    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9c8q4"]
Oct 31 00:28:50.889633 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.889605    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9c8q4"]
Oct 31 00:28:50.889782 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.889703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:50.905280 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.905191    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7wwd7\""
Oct 31 00:28:50.905522 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.905502    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:28:50.905646 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.905614    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:28:50.905788 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.905776    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:28:50.905974 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.905953    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:28:50.921536 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.921505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd577\" (UniqueName: \"kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577\") pod \"migrator-66df5d496b-429vc\" (UID: \"f4c055f3-44ac-423d-9c0c-85383b5d8d94\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:50.932143 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.932111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.935596 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.935571    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd577\" (UniqueName: \"kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577\") pod \"migrator-66df5d496b-429vc\" (UID: \"f4c055f3-44ac-423d-9c0c-85383b5d8d94\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:50.965204 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:50.965181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:51.022015 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.021987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdf6\" (UniqueName: \"kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.022128 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.022109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.022184 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.022170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.123340 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.123261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.123340 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.123333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.123513 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.123454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdf6\" (UniqueName: \"kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.124035 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.124005    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.126348 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.126328    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.141849 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.141824    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdf6\" (UniqueName: \"kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.227551 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.227519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:51.269524 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.269450    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"]
Oct 31 00:28:51.320920 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.320884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:51.325492 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.325465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:51.325633 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.325615    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:51.325718 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.325704    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.325666935 +0000 UTC m=+38.607699940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:51.370624 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:51.370595    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c055f3_44ac_423d_9c0c_85383b5d8d94.slice/crio-523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565 WatchSource:0}: Error finding container 523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565: Status 404 returned error can't find the container with id 523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565
Oct 31 00:28:51.501827 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.501789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:51.528318 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.528289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:51.528427 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.528400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:51.528549 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.528526    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:51.528652 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.528585    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.528567542 +0000 UTC m=+36.810600554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:51.528892 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.528878    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:51.528951 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:51.528915    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.528906242 +0000 UTC m=+36.810939253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:51.562247 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.562212    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc" event={"ID":"f4c055f3-44ac-423d-9c0c-85383b5d8d94","Type":"ContainerStarted","Data":"523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565"}
Oct 31 00:28:51.565538 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.565506    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6kkm" event={"ID":"3625a6a6-66db-49b1-8fb6-fae35972b557","Type":"ContainerStarted","Data":"651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5"}
Oct 31 00:28:51.568797 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.568769    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jrkck" event={"ID":"c3e796e1-0c4b-455c-906d-98ee4c141fe0","Type":"ContainerStarted","Data":"e97bef9029ebd211163e9499ca69091bf8e68b4294182d00f140877c1ae17325"}
Oct 31 00:28:51.589020 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.588973    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jrkck" podStartSLOduration=1.3260706070000001 podStartE2EDuration="4.588954744s" totalImagesPullingTime="3.262884137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:51.588712533 +0000 UTC m=+34.870745551" watchObservedRunningTime="2026-10-31 00:28:51.588954744 +0000 UTC m=+34.870987763"
Oct 31 00:28:51.687914 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:51.687880    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9c8q4"]
Oct 31 00:28:51.705858 ip-10-0-135-193 kubenswrapper[2582]: W1031 00:28:51.705825    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48764144_837f_4ae6_ac64_982b8c04c362.slice/crio-90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e WatchSource:0}: Error finding container 90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e: Status 404 returned error can't find the container with id 90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e
Oct 31 00:28:52.271297 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:52.271261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:52.576106 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:52.576027    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4" event={"ID":"48764144-837f-4ae6-ac64-982b8c04c362","Type":"ContainerStarted","Data":"90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e"}
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:53.546877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:53.546991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:53.547126    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:53.547195    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:57.54717625 +0000 UTC m=+40.829209264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:53.547264    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:53.547586 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:53.547328    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm podName:bf22e752-9631-48b6-94da-3e7c458d7601 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:57.547309325 +0000 UTC m=+40.829342337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnddf" (UID: "bf22e752-9631-48b6-94da-3e7c458d7601") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:53.581598 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:53.581554    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc" event={"ID":"f4c055f3-44ac-423d-9c0c-85383b5d8d94","Type":"ContainerStarted","Data":"ea36b96ac5e50f6187554675f135ffb8246e4ea96ac0ab66095e923c2cb0ffcb"}
Oct 31 00:28:53.601715 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:53.601642    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc" podStartSLOduration=1.741759099 podStartE2EDuration="3.601622085s" totalImagesPullingTime="1.859862986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:53.600229697 +0000 UTC m=+36.882262719" watchObservedRunningTime="2026-10-31 00:28:53.601622085 +0000 UTC m=+36.883655103"
Oct 31 00:28:54.587773 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:54.587735    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" event={"ID":"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971","Type":"ContainerStarted","Data":"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"}
Oct 31 00:28:54.609595 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:54.609531    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" podStartSLOduration=2.08240127 podStartE2EDuration="5.609512139s" totalImagesPullingTime="3.527110869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:54.608946479 +0000 UTC m=+37.890979499" watchObservedRunningTime="2026-10-31 00:28:54.609512139 +0000 UTC m=+37.891545177"
Oct 31 00:28:55.001932 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:55.001890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:55.365916 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:55.365833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:55.366072 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:55.366015    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:55.366117 ip-10-0-135-193 kubenswrapper[2582]: E1031 00:28:55.366096    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:03.36607592 +0000 UTC m=+46.648108934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:55.594647 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:55.594607    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4" event={"ID":"48764144-837f-4ae6-ac64-982b8c04c362","Type":"ContainerStarted","Data":"f16c1b3a5fb5fc1a57466f9f761ab52b348c7ffa3998683b097416ef8d774d94"}
Oct 31 00:28:55.615071 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:55.615018    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4" podStartSLOduration=2.185979734 podStartE2EDuration="5.6150025s" totalImagesPullingTime="3.429022766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:55.614856882 +0000 UTC m=+38.896889901" watchObservedRunningTime="2026-10-31 00:28:55.6150025 +0000 UTC m=+38.897035522"
Oct 31 00:28:55.766192 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:55.766151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:56.260344 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.260151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:56.359664 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.359638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ck7z7_22e47f56-d809-4b6a-861b-98b7bd8241fb/dns-node-resolver/0.log"
Oct 31 00:28:56.400695 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.400647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:56.547124 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.547061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:56.598553 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.598519    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6kkm" event={"ID":"3625a6a6-66db-49b1-8fb6-fae35972b557","Type":"ContainerStarted","Data":"25dad5cc20595dd704d15df5202feb30cae92d5c9266623810414abcfa2787c9"}
Oct 31 00:28:56.617496 ip-10-0-135-193 kubenswrapper[2582]: I1031 00:28:56.617447    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c6kkm" podStartSLOduration=34.803828635 podStartE2EDuration="39.617434154s" totalImagesPullingTime="4.813605519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:56.616985455 +0000 UTC m=+39.899018476" watchObservedRunningTime="2026-10-31 00:28:56.617434154 +0000 UTC m=+39.899467173"
Oct 31 00:28:56.987061 ip-10-0-135-193 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:28:57.003072 ip-10-0-135-193 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:28:57.003240 ip-10-0-135-193 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:28:57.003285 ip-10-0-135-193 systemd[1]: kubelet.service: Consumed 2.410s CPU time.
Oct 31 00:28:57.014337 ip-10-0-135-193 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:28:57.119976 ip-10-0-135-193 kubenswrapper[4553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:57.119976 ip-10-0-135-193 kubenswrapper[4553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:28:57.119976 ip-10-0-135-193 kubenswrapper[4553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:57.119976 ip-10-0-135-193 kubenswrapper[4553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:57.121997 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.121923    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.121997 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.121997    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.121997 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122001    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122005    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122008    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122011    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122014    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122016    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122021    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122025    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122028    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122030    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122033    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122036    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122038    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122041    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122044    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122047    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122049    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122052    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122054    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.122100 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122057    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122060    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122062    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122064    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122067    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122070    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122072    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122075    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122078    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122080    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122083    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122087    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122090    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122092    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122095    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122098    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122100    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122103    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122106    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.122579 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122109    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122112    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122114    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122117    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122119    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122121    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122124    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122127    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122129    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122132    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122134    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122138    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122141    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122143    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122145    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122148    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122150    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122153    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122155    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122158    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.123083 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122160    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122163    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122165    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122168    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122170    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122174    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122176    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122179    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122181    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122184    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122186    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122189    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122192    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122194    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122197    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122199    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122202    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122204    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122207    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.123626 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122209    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122213    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122217    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122219    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122224    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122226    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122229    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122231    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122234    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122236    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122239    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122241    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122243    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122246    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122248    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122251    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122253    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122256    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122259    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.124134 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122262    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122264    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122267    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122270    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122273    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122276    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122278    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.122280    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123097    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123104    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123107    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123111    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123113    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123116    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123119    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123121    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123124    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123127    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123130    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123132    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.124611 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123136    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123138    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123141    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123143    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123147    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123150    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123153    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123155    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123158    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123161    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123165    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123169    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123173    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123178    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123182    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123186    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123190    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123194    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123198    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.125139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123205    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123209    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123212    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123215    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123217    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123220    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123222    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123225    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123227    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123230    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123232    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123235    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123237    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123240    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123242    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123244    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123247    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123250    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123252    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123255    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.125615 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123257    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123260    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123263    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123266    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123269    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123271    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123274    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123277    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123282    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123285    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123289    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123291    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123294    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123297    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123300    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123303    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123305    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123308    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123311    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.126148 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123313    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123315    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123318    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123320    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123323    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123325    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123327    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123331    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123333    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123336    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123338    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123341    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123343    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123346    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123348    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123351    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123353    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123356    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123358    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.126655 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123361    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123363    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123366    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123368    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123371    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123373    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123376    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123378    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123381    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123383    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123385    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123388    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123390    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123393    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123395    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123398    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123400    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123885    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123891    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.127153 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123895    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123898    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123901    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123903    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123906    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123908    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123911    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123914    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123916    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123918    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123921    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123923    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123927    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123931    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123934    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123937    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123940    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123942    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123945    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.127619 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123948    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123951    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123953    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123956    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123958    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123960    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123963    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123965    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123968    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123970    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123973    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123975    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123978    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123980    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123983    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123986    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123988    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123991    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123993    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123995    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.128191 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.123998    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124000    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124003    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124005    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124008    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124010    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124013    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124016    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124018    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124021    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124024    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124027    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124029    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124031    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124034    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124036    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124039    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124041    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124043    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124046    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.129030 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124049    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124051    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124054    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124056    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124059    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124061    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124064    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124066    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124069    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124071    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124074    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124076    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124079    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124081    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124084    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124086    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124089    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124091    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124093    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124096    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.129840 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124098    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124101    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124104    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124106    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124109    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124111    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124114    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124116    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124119    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124121    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124124    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124126    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124129    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124131    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124134    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124136    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124139    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124142    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124144    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.130463 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124147    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124149    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124153    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124156    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124158    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124161    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124252    4553 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124267    4553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124273    4553 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124277    4553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124281    4553 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124284    4553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124289    4553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124293    4553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124298    4553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124301    4553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124305    4553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124309    4553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124312    4553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124314    4553 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124317    4553 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124320    4553 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124323    4553 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:28:57.130952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124326    4553 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124331    4553 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124335    4553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124338    4553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124341    4553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124345    4553 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124348    4553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124351    4553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124354    4553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124358    4553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124361    4553 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124364    4553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124367    4553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124370    4553 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124372    4553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124377    4553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124380    4553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124382    4553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124385    4553 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124388    4553 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124391    4553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124395    4553 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124398    4553 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124403    4553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:28:57.131543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124406    4553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124409    4553 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124414    4553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124417    4553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124420    4553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124423    4553 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124426    4553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124429    4553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124432    4553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124435    4553 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124439    4553 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124442    4553 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124445    4553 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124449    4553 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124452    4553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124455    4553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124458    4553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124461    4553 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124464    4553 flags.go:64] FLAG: --help="false"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124467    4553 flags.go:64] FLAG: --hostname-override="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124470    4553 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124473    4553 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124476    4553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:28:57.132232 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124479    4553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124483    4553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124486    4553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124488    4553 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124491    4553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124494    4553 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124497    4553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124500    4553 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124506    4553 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124509    4553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124512    4553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124515    4553 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124523    4553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124526    4553 flags.go:64] FLAG: --lock-file=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124529    4553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124532    4553 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124536    4553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124545    4553 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124548    4553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124552    4553 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124555    4553 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124558    4553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124561    4553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124564    4553 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124567    4553 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:28:57.132804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124571    4553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124574    4553 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124578    4553 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124581    4553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124584    4553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124587    4553 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124590    4553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124593    4553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124596    4553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124599    4553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124605    4553 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124608    4553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124612    4553 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124615    4553 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124618    4553 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124622    4553 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124625    4553 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124628    4553 flags.go:64] FLAG: --port="10250"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124631    4553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124634    4553 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0423b6e8d51d21c41"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124643    4553 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124646    4553 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124649    4553 flags.go:64] FLAG: --register-node="true"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124652    4553 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124656    4553 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124658    4553 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:28:57.133434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124662    4553 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124665    4553 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124669    4553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124686    4553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124689    4553 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124692    4553 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124695    4553 flags.go:64] FLAG: --runonce="false"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124698    4553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124701    4553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124704    4553 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124707    4553 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124709    4553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124719    4553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124722    4553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124725    4553 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124727    4553 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124730    4553 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124733    4553 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124736    4553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124739    4553 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124742    4553 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124746    4553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124751    4553 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124755    4553 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124757    4553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:28:57.134096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124766    4553 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124769    4553 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124772    4553 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124775    4553 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124778    4553 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124781    4553 flags.go:64] FLAG: --v="2"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124785    4553 flags.go:64] FLAG: --version="false"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124790    4553 flags.go:64] FLAG: --vmodule=""
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124794    4553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.124797    4553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124944    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124949    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124953    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124956    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124958    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124961    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124964    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124966    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124969    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124971    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124974    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.134807 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124976    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124979    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124981    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124984    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124986    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124989    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124991    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124995    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.124997    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125000    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125002    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125005    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125007    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125010    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125013    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125016    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125018    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125021    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125025    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.135453 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125028    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125030    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125032    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125035    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125037    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125040    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125042    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125045    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125048    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125050    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125053    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125055    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125057    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125060    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125062    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125065    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125067    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125069    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125072    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.136143 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125076    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125081    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125085    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125087    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125090    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125093    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125096    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125098    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125101    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125110    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125113    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125115    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125120    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125123    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125126    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125128    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125130    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125133    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125136    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.136667 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125138    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125141    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125143    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125146    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125149    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125152    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125154    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125157    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125159    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125162    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125165    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125167    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125170    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125172    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125176    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125178    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125181    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125183    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125186    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125188    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.137240 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125191    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125193    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125195    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125198    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125201    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125205    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125207    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125210    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125212    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125215    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125217    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125220    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125222    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125225    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125228    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125230    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125233    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.137801 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125235    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.125262    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125299    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125303    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125306    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125309    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125313    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125315    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125318    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125322    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125324    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125327    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125329    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125332    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125334    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125336    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.138376 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125339    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125341    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125345    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125349    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125352    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125357    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125360    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125363    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125365    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125368    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125371    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125373    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125375    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125378    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125380    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125383    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125385    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125388    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125391    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.138954 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125393    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125395    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125398    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125401    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125403    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125406    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125410    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125412    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125415    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125417    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125419    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125422    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125425    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125427    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125430    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125432    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125435    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125438    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125441    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.139492 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125444    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125446    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125448    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125451    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125453    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125456    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125458    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125461    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125463    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125466    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125468    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125471    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125473    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125476    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125478    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125481    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125483    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125486    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125488    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125492    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.139987 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125494    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125497    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125500    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125504    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125507    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125509    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125512    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125515    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125517    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125519    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125522    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125526    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125529    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125532    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125534    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125536    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125539    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125541    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125544    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.140482 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125546    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125549    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125551    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125554    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125556    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125559    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125561    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125564    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125566    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125568    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125572    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125574    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125578    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125581    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.125583    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.130629    4553 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.130651    4553 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.130687    4553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.130692    4553 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130750    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.140955 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130754    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130758    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130760    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130763    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130766    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130768    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130771    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130774    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130776    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130779    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130781    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130784    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130786    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130793    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130796    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130799    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130801    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130804    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130807    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.141451 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130810    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130812    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130814    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130818    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130820    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130823    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130825    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130828    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130831    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130834    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130836    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130840    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130844    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130847    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130850    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130853    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130855    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130858    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130860    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130863    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.141914 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130865    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130868    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130870    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130873    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130875    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130878    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130881    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130884    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130887    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130890    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130892    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130895    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130898    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130901    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130903    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130906    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130908    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130911    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130913    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130916    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.142401 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130919    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130921    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130924    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130926    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130929    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130931    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130934    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130937    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130939    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130942    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130944    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130947    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130949    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130951    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130954    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130957    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130960    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130962    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130965    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.142921 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130968    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130971    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130974    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130976    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130978    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130981    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130984    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130986    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130989    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130991    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130994    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130997    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.130999    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131001    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131004    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131006    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131010    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131014    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131016    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.143397 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131019    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131021    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131024    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131026    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131029    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131031    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131033    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131036    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.131063    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131196    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131200    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131203    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131205    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131208    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131211    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131214    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.144093 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131216    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131219    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131222    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131224    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131227    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131230    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131232    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131234    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131237    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131239    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131241    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131244    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131246    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131248    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131251    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131253    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131256    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131258    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131261    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131263    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.144884 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131266    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131268    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131270    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131273    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131275    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131277    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131281    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131285    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131288    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131290    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131293    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131296    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131299    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131301    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131304    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131307    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131309    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131311    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131314    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.145514 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131316    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131319    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131321    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131324    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131326    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131329    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131331    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131333    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131336    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131339    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131341    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131343    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131346    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131349    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131351    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131353    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131357    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131361    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131364    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.146222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131366    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131369    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131371    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131374    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131377    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131379    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131383    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131385    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131388    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131391    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131393    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131396    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131398    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131400    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131403    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131405    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131408    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131410    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131412    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131415    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.146950 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131417    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131419    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131422    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131424    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131427    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131429    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131431    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131434    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131437    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131439    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131442    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131444    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131447    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131449    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131452    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131456    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131460    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131463    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131468    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.147648 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131472    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131477    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.131509    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131552    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131558    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131562    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131564    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131567    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131570    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131572    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131575    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131578    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131580    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131583    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131585    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131588    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:57.148139 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131591    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131593    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131597    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131600    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131603    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131606    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131608    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131611    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131613    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131616    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131618    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131620    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131623    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131626    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131628    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131631    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131633    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131636    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131638    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:57.148585 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131642    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131644    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131647    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131649    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131652    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131654    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131656    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131659    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131661    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131663    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131692    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131696    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131698    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131701    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131708    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131710    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131713    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131715    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131718    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:57.149102 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131721    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131724    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131727    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131729    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131732    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131734    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131737    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131740    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131742    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131745    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131747    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131750    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131754    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131757    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131759    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131762    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131764    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131767    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131769    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131772    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:57.149928 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131774    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131777    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131779    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131782    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131784    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131786    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131789    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131792    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131795    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131798    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131800    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131803    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131805    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131808    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131810    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131813    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131815    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131818    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131820    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131823    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:57.150799 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131826    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131830    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131833    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131836    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131839    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131843    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131846    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131848    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131851    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131853    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131856    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131858    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131861    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131863    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:57.131867    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.131925    4553 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.132919    4553 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.132973    4553 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:28:57.151658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.133149    4553 server.go:1014] "Starting client certificate rotation"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.133333    4553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.133395    4553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:17 +0000 UTC" deadline="2028-06-14 05:29:43.786159992 +0000 UTC"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.133452    4553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14213h0m46.652709564s"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.133636    4553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.134706    4553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.136217    4553 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.137958    4553 log.go:25] "Validated CRI v1 image API"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.138778    4553 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:28:57.152463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.140802    4553 fs.go:135] Filesystem UUIDs: map[1fd60f1f-eff5-46ee-8004-3ee3bad04174:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bb45d447-1e84-48b3-b4a2-7d85c8f04e91:/dev/nvme0n1p3]
Oct 31 00:28:57.152947 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.140818    4553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b2c563768268d3226f232fc7dbff8d744160f0aa235d38418d788f17d995ffa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b2c563768268d3226f232fc7dbff8d744160f0aa235d38418d788f17d995ffa/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~projected/kube-api-access-sswmc:{mountpoint:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~projected/kube-api-access-sswmc major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22e47f56-d809-4b6a-861b-98b7bd8241fb/volumes/kubernetes.io~projected/kube-api-access-kksbs:{mountpoint:/var/lib/kubelet/pods/22e47f56-d809-4b6a-861b-98b7bd8241fb/volumes/kubernetes.io~projected/kube-api-access-kksbs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f0fe6f-7a3d-4966-95be-31c2ee3941a6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/25f0fe6f-7a3d-4966-95be-31c2ee3941a6/volumes/kubernetes.io~secret/agent-certs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0/volumes/kubernetes.io~projected/kube-api-access-qwngb:{mountpoint:/var/lib/kubelet/pods/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0/volumes/kubernetes.io~projected/kube-api-access-qwngb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3625a6a6-66db-49b1-8fb6-fae35972b557/volumes/kubernetes.io~projected/kube-api-access-h44ts:{mountpoint:/var/lib/kubelet/pods/3625a6a6-66db-49b1-8fb6-fae35972b557/volumes/kubernetes.io~projected/kube-api-access-h44ts major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971/volumes/kubernetes.io~projected/kube-api-access-j9mjd:{mountpoint:/var/lib/kubelet/pods/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971/volumes/kubernetes.io~projected/kube-api-access-j9mjd major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~projected/kube-api-access-mhdf6:{mountpoint:/var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~projected/kube-api-access-mhdf6 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~secret/signing-key major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d383848-ee88-4158-b9f8-8dd88fdab4c4/volumes/kubernetes.io~projected/kube-api-access-h47sq:{mountpoint:/var/lib/kubelet/pods/5d383848-ee88-4158-b9f8-8dd88fdab4c4/volumes/kubernetes.io~projected/kube-api-access-h47sq major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd8b314-8c3e-45a3-9395-bd88d51d1b08/volumes/kubernetes.io~projected/kube-api-access-jdf44:{mountpoint:/var/lib/kubelet/pods/9cd8b314-8c3e-45a3-9395-bd88d51d1b08/volumes/kubernetes.io~projected/kube-api-access-jdf44 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a862323a-20e5-41f7-86bf-79f2964d1ec5/volumes/kubernetes.io~projected/kube-api-access-dgjrh:{mountpoint:/var/lib/kubelet/pods/a862323a-20e5-41f7-86bf-79f2964d1ec5/volumes/kubernetes.io~projected/kube-api-access-dgjrh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed308ea-c7aa-4772-ab2e-458e6ef7a33d/volumes/kubernetes.io~projected/kube-api-access-hc6nb:{mountpoint:/var/lib/kubelet/pods/bed308ea-c7aa-4772-ab2e-458e6ef7a33d/volumes/kubernetes.io~projected/kube-api-access-hc6nb major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf22e752-9631-48b6-94da-3e7c458d7601/volumes/kubernetes.io~projected/kube-api-access-fc9rk:{mountpoint:/var/lib/kubelet/pods/bf22e752-9631-48b6-94da-3e7c458d7601/volumes/kubernetes.io~projected/kube-api-access-fc9rk major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~projected/kube-api-access-gdwsw:{mountpoint:/var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~projected/kube-api-access-gdwsw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/tmp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~projected/kube-api-access-bkd56:{mountpoint:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~projected/kube-api-access-bkd56 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ad6861-24a8-40ff-8a4e-ee332a213b54/volumes/kubernetes.io~projected/kube-api-access-h75gr:{mountpoint:/var/lib/kubelet/pods/c6ad6861-24a8-40ff-8a4e-ee332a213b54/volumes/kubernetes.io~projected/kube-api-access-h75gr major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfee234-053b-4291-aecd-5c197dded041/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cbfee234-053b-4291-aecd-5c197dded041/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5ba2243-f40b-4f95-a955-bd82c885d457/volumes/kubernetes.io~projected/kube-api-access-8wtjg:{mountpoint:/var/lib/kubelet/pods/e5ba2243-f40b-4f95-a955-bd82c885d457/volumes/kubernetes.io~projected/kube-api-access-8wtjg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4c055f3-44ac-423d-9c0c-85383b5d8d94/volumes/kubernetes.io~projected/kube-api-access-kd577:{mountpoint:/var/lib/kubelet/pods/f4c055f3-44ac-423d-9c0c-85383b5d8d94/volumes/kubernetes.io~projected/kube-api-access-kd577 major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1420efda5d52aa16aa4cefa33b11bfa4a82cb381e5005a8a111456330ced0cd1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/340e7d3211481e058daffafa9a5f75d8fe2e89d9bbe7beb1fc733982ddfbebb2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7c0da15d04b61961b3bc42edfa13d5ba3dff64e82871eb6adf5d457e07f0dd82/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/04331deee4f4a42f50cf41cb5d2ee2a9a86233bc6a731f2b65f1878bc17344aa/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/039b8308b11ed39edfd71887d955174d1b1d2224b0bed0845dfd794fb8f67cbc/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e82b96fc65f83e564929e538e5fff72767403bd4d07b9b731aa79f3613113a36/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1846fd784ca4c14a21180144e37a95d8d9a5d7e2ccf8cc1332c8baaab7abb190/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/75cb56cbc3d29558ff81e1ce852f3fa90a7fe63ac08e88238508270fdaa308c9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/73833dc9deb08eb25a5436befae7515292cf4ad32f676ace5d5ae7c33fa69db2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/177d9058d8036b8a247de481ab06e3e408f38bd142d1bdb2b48ad8017c234e7a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/61ea627c08682ca675a6a2aabcdd536aa4798cdd60020d0dd4bfe2a74ca9b998/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/eb43a62b446c826c12ad02197caf54954014d59128303064123b259f88a1bb5c/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/aa46079214a0a2ce9cd7eb13f0e0e66e8f7cb37ccf3687a71bdcc8c87f3ce3c6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0e27160f6ebda496adf6e0e2530acb24782c4ae343b8e27599cf8c666311d4c8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/9444fd0e3c0cb1e076768a16d13733e7220d1d9e989ec1b3051c6491821fb401/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b965051ce76d9c40bc234b77e0a74b9a8fd0e6d5bbba71ec03fbe30c0c0e3b63/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ee59346499d6db22ff96dad7c8d3885c812677add04b0d0a86e5e638904608c1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/289f8a5cbd72844574d3f269ca7adb4f4b04499ae97c6d19758e3879e0a0c499/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/2dee3f6f6a83dfe2aa97b3b50bd42feaa3175f4b27e80eb70b9a8fdba3626245/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/088a9c64c90c7a1ac539f2158dbe693d07220e402127ec47be73c783631c5c96/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/851eb2d1c126a89766cb99b8893dc16b6e86c7b2b229f7c34ac2546c33c0b9a6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/6d91243b30748e376461dd8ed9d57461fc85caf7a9eb79d0637e25b5ce59912b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/f216d1887c3470c3ddfe28dc6007191316ee13683ef97176acbbceb4a5c3340f/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/b5d02036b7b442f8915f3a693050f02f6f14e451d28276a1f2aef62c1a3502f0/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/3cf8ceddeeaffd8e5a803474f5647963caed48ff29b31926c7537fde848b6d14/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/ed48cfb928573a15b1405ae4325a22524d9695634f1d4f59da9c64c2e36d95f2/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/2297a5a753e1dd1316d063755d10098015483731974bed9a47533f553383b396/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/0de0f0b884586d9ddfaa79bd30b0bb674cd1e7e6e7131ddb1e3cc5b8b338a2fd/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/f127d75cea4c924282e01d35ff35e113a229953ad00fef55ddfd55676dbcddf9/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/ea9a56773ee013fc38b589b6388cbd1917dc83e21c5fa04c3be3b1c442e37750/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d9967ac9589c7d653eaaa42e0052e46786d3c8c265800b564a08ff56522caf0c/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/1c931e07594e99136eefa5a8a74e1a753ffbb05fc8be28da0b2e28d3b2d3b291/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/559df49da59d9340cf4e1566dc94bbaa2ed6e1ccb1d10ef8b88486e06ea53753/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/ff5c35071d32376fc6e55c2fd36217e3e149d01ec4ccc82567cbba19d46b0c45/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/76fab52f52da6d09568b045e7838630ca82abf603c573eaeabd22e134ed7bb4a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f67f86e8396c0ff138e9be6457eaf69e8fec5471d4f71a2360a9c8e110914afe/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c59232cc935bfd2e901091ccf5dc736a64ef84c0a0d7c14c534410088e011764/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1231bdacdfce53d5ddfe473f714402cb859d768da0d4c4031194b5a9b9af99ca/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/89c4af1cd84cd72c2b0c6cc2041f943f22f8cdf6ae76ef054d3c35f2e6aad83d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/381454e0a2a400fc1b57b00dd8299fc4d5ff97d4d62a0aa2853fc88119f6ea09/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/08798073e2724a6d090bdf5e163820a2910d62e507db5da5681e7310377058a6/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3c392ae2889c930c57cf85105010083d2169b0732ad492b6db06a52a322901d3/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/ab922003b8a1159a5d7f79f8b9e5e08fc8305576dc3a57e34a1889e09ad6f5ae/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.147945    4553 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:57.147466847 +0000 UTC m=+0.056469492 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121791 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247ef87707fd58390dcbb11c7e99d9 SystemUUID:ec247ef8-7707-fd58-390d-cbb11c7e99d9 BootID:a4ca3629-3cf9-4ad5-84a8-7ee6bfe48d5e Filesystems:[{Device:/run/containers/storage/overlay-containers/9e47b528ec33c9025be4e53ce9365c92c33dbcf13747055df2838dbff02e9d08/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfee234-053b-4291-aecd-5c197dded041/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bf22e752-9631-48b6-94da-3e7c458d7601/volumes/kubernetes.io~projected/kube-api-access-fc9rk DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae130f26be53f678d284c5618bdda4bf260f9eff1a830bc55f9c41acd4c203d2/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971/volumes/kubernetes.io~projected/kube-api-access-j9mjd DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4c055f3-44ac-423d-9c0c-85383b5d8d94/volumes/kubernetes.io~projected/kube-api-access-kd577 DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9d6ccc9a09f88cb67606e61b5d19d70e9b6f5573309de82586f3d7b84e7d076/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/651223adc7bf5c02344e005bcab7ddcd8d7ae00ecb5791872537f2c77ebaa5d5/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~projected/kube-api-access-bkd56 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a862323a-20e5-41f7-86bf-79f2964d1ec5/volumes/kubernetes.io~projected/kube-api-access-dgjrh DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8832c290751c0e7724b0999dd79f40c810dd0d25a3fcdaa961ceaf495e4e9d24/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~projected/kube-api-access-gdwsw DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae67b72d09b31d00a13a71acab0b55fb787ad78ce890281ff52cc9b472861ef6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd8b314-8c3e-45a3-9395-bd88d51d1b08/volumes/kubernetes.io~projected/kube-api-access-jdf44 DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0/volumes/kubernetes.io~projected/kube-api-access-qwngb DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b1b7360a7ad75f86491fd8d92b31ba38e5b8d5d9e418b054f7a4297b8952a77/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~projected/kube-api-access-mhdf6 DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/523c6c071687bcede1b7e0c7572661e0571a0163ccc3d01dd23314fddecc6565/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/48764144-837f-4ae6-ac64-982b8c04c362/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22e47f56-d809-4b6a-861b-98b7bd8241fb/volumes/kubernetes.io~projected/kube-api-access-kksbs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d383848-ee88-4158-b9f8-8dd88fdab4c4/volumes/kubernetes.io~projected/kube-api-access-h47sq DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/bed308ea-c7aa-4772-ab2e-458e6ef7a33d/volumes/kubernetes.io~projected/kube-api-access-hc6nb DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25f0fe6f-7a3d-4966-95be-31c2ee3941a6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee7fc5ac392f9ee35c789fd145d7c88657590f1402063636bfc0e657c7445e39/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3625a6a6-66db-49b1-8fb6-fae35972b557/volumes/kubernetes.io~projected/kube-api-access-h44ts DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volumes/kubernetes.io~projected/kube-api-access-sswmc DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ad6861-24a8-40ff-8a4e-ee332a213b54/volumes/kubernetes.io~projected/kube-api-access-h75gr DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c61a812b-5c90-419b-bf1d-6c2cad75af52/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b2c563768268d3226f232fc7dbff8d744160f0aa235d38418d788f17d995ffa/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0a7940dcc94a58a08381cd0801c668ed28de55bed547497dda694184a6a646a/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e79d059-a2d5-419e-bb3a-de74bb0082e5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e5ba2243-f40b-4f95-a955-bd82c885d457/volumes/kubernetes.io~projected/kube-api-access-8wtjg DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90a96ab67fac4af6f9bf75541469e89e048860eaf992de0540d92efa6399ad7e/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e796e1-0c4b-455c-906d-98ee4c141fe0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/035fb308ef42ac87fde96e501bc7bb2a9dd0456425ec76a48041a61278433e4d/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:035fb308ef42ac8 MacAddress:2a:43:2c:58:d4:74 Speed:10000 Mtu:8901} {Name:155349b096d8dff MacAddress:fa:84:71:70:95:24 Speed:10000 Mtu:8901} {Name:523c6c071687bce MacAddress:9e:13:1c:c6:02:22 Speed:10000 Mtu:8901} {Name:651223adc7bf5c0 MacAddress:76:48:8e:e2:2c:8a Speed:10000 Mtu:8901} {Name:90a96ab67fac4af MacAddress:7e:c8:28:39:2e:6b Speed:10000 Mtu:8901} {Name:a9d6ccc9a09f88c MacAddress:5a:54:19:45:0e:19 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ef:45:af:db Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:0b:88:a0:3d:c8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ef:45:af:db Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:1a:15:88:86:cb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:85:61:3d:60:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148234    4553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148276    4553 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148533    4553 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148560    4553 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-193.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148692    4553 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148700    4553 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148714    4553 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148727    4553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148805    4553 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.148950    4553 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149048    4553 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149080    4553 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149089    4553 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149098    4553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149106    4553 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149113    4553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.149984    4553 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.150002    4553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.150580    4553 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.152635    4553 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.152645    4553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.152765    4553 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.152972    4553 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153001    4553 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153099    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153116    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153126    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153133    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153140    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153147    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153166    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153175    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153183    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153191    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153202    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:28:57.153216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153216    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:28:57.154865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153277    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:28:57.154865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.153283    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:28:57.157041 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.157025    4553 server.go:1263] "Started kubelet"
Oct 31 00:28:57.157204 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.157122    4553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:28:57.157263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.157234    4553 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:28:57.157562 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.157535    4553 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:28:57.157622 ip-10-0-135-193 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:28:57.157787 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.157626    4553 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.161510    4553 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.161715    4553 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.161926    4553 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.161944    4553 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.162082    4553 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-193.us-east-2.compute.internal\" not found"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.162153    4553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.162270    4553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.162326    4553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:18 +0000 UTC" deadline="2028-07-14 20:43:17.579840054 +0000 UTC"
Oct 31 00:28:57.164131 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.162611    4553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14948h14m20.417237219s"
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164330    4553 factory.go:55] Registering systemd factory
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164346    4553 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164787    4553 factory.go:153] Registering CRI-O factory
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164807    4553 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164872    4553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164893    4553 factory.go:103] Registering Raw factory
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.164916    4553 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:28:57.166616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.165300    4553 manager.go:319] Starting recovery of all containers
Oct 31 00:28:57.167577 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167541    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e79d059-a2d5-419e-bb3a-de74bb0082e5" volumeName="kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167578    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61a812b-5c90-419b-bf1d-6c2cad75af52" volumeName="kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167593    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" volumeName="kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167606    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" volumeName="kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167618    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d383848-ee88-4158-b9f8-8dd88fdab4c4" volumeName="kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167632    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" volumeName="kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167644    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e796e1-0c4b-455c-906d-98ee4c141fe0" volumeName="kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert" seLinuxMountContext=""
Oct 31 00:28:57.167660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167654    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ad6861-24a8-40ff-8a4e-ee332a213b54" volumeName="kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167692    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4c055f3-44ac-423d-9c0c-85383b5d8d94" volumeName="kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167705    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" volumeName="kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167716    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ba2243-f40b-4f95-a955-bd82c885d457" volumeName="kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167728    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f0fe6f-7a3d-4966-95be-31c2ee3941a6" volumeName="kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167739    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f0fe6f-7a3d-4966-95be-31c2ee3941a6" volumeName="kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167777    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa170b9-6ca8-45d3-b0a0-7394bf27fff0" volumeName="kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167793    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf22e752-9631-48b6-94da-3e7c458d7601" volumeName="kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167805    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ba2243-f40b-4f95-a955-bd82c885d457" volumeName="kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167817    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e79d059-a2d5-419e-bb3a-de74bb0082e5" volumeName="kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167841    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3625a6a6-66db-49b1-8fb6-fae35972b557" volumeName="kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167853    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" volumeName="kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167866    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e79d059-a2d5-419e-bb3a-de74bb0082e5" volumeName="kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167880    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e79d059-a2d5-419e-bb3a-de74bb0082e5" volumeName="kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167891    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22e47f56-d809-4b6a-861b-98b7bd8241fb" volumeName="kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167901    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d383848-ee88-4158-b9f8-8dd88fdab4c4" volumeName="kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167916    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a862323a-20e5-41f7-86bf-79f2964d1ec5" volumeName="kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167929    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61a812b-5c90-419b-bf1d-6c2cad75af52" volumeName="kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167943    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ad6861-24a8-40ff-8a4e-ee332a213b54" volumeName="kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167956    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ad6861-24a8-40ff-8a4e-ee332a213b54" volumeName="kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167970    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" volumeName="kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.167988    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48764144-837f-4ae6-ac64-982b8c04c362" volumeName="kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168007    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed308ea-c7aa-4772-ab2e-458e6ef7a33d" volumeName="kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168037    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed308ea-c7aa-4772-ab2e-458e6ef7a33d" volumeName="kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb" seLinuxMountContext=""
Oct 31 00:28:57.168063 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168066    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e796e1-0c4b-455c-906d-98ee4c141fe0" volumeName="kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168083    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61a812b-5c90-419b-bf1d-6c2cad75af52" volumeName="kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168098    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfee234-053b-4291-aecd-5c197dded041" volumeName="kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168109    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" volumeName="kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168144    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22e47f56-d809-4b6a-861b-98b7bd8241fb" volumeName="kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168156    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48764144-837f-4ae6-ac64-982b8c04c362" volumeName="kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168169    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48764144-837f-4ae6-ac64-982b8c04c362" volumeName="kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168195    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d383848-ee88-4158-b9f8-8dd88fdab4c4" volumeName="kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168206    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" volumeName="kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168234    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf22e752-9631-48b6-94da-3e7c458d7601" volumeName="kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168251    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e79d059-a2d5-419e-bb3a-de74bb0082e5" volumeName="kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc" seLinuxMountContext=""
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168264    4553 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:28:57.169183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.168272    4553 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:28:57.171103 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.171084    4553 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:57.173030 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.173006    4553 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:57.174867 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.174848    4553 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:57.176706 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.175858    4553 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:28:57.177645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.177615    4553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:28:57.187027 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.187009    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:57.187214 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.187196    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:57.194006 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.193983    4553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:28:57.194006 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.194007    4553 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:28:57.194156 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.194026    4553 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:28:57.194156 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.194067    4553 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:28:57.200766 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.200744    4553 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:57.212604 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.212583    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:28:57.212989 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.212962    4553 generic.go:356] "Generic (PLEG): container finished" podID="0e79d059-a2d5-419e-bb3a-de74bb0082e5" containerID="7bb4d212ab315d177657914c406cc154c82ab5e19686e70fa8403b85f46cb363" exitCode=1
Oct 31 00:28:57.216848 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216821    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="b855ed38811b18a9ec2958a06124d893a3977a4410af9cad71acdcf7c9ae25df" exitCode=0
Oct 31 00:28:57.216937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216849    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="076728c21752f8de1cad538b2630d5cb23b084cc944af00963e8cc23b55d61c9" exitCode=0
Oct 31 00:28:57.216937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216859    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="9f6b6ae756d29aae2b54e337a608d662fc1a70a92a2b4bb9738286fc75bfec72" exitCode=0
Oct 31 00:28:57.216937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216868    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="d1e0f36cefe71617c71f22b5b67823dc38d3e32f53fc80ee6c059266e5bff5f9" exitCode=0
Oct 31 00:28:57.216937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216876    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="58fcba4e12f3e6addbd27e0eb9973badfdd60d0ed833625021e5466304a49898" exitCode=0
Oct 31 00:28:57.216937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.216884    4553 generic.go:356] "Generic (PLEG): container finished" podID="9cd8b314-8c3e-45a3-9395-bd88d51d1b08" containerID="92fdffe787acc874f8e4fce3afe3393b90a0806752bc5e485ab13a907813d720" exitCode=0
Oct 31 00:28:57.218770 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.218749    4553 generic.go:356] "Generic (PLEG): container finished" podID="f9318895ca5d45d45cb102c90334bd92" containerID="8e6a1946ab4519ad5233c9072ed090c769e3867187471f0dbba786f4e6e79c68" exitCode=0
Oct 31 00:28:57.242728 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.242710    4553 manager.go:324] Recovery completed
Oct 31 00:28:57.258996 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.258974    4553 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:28:57.258996 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.258992    4553 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:28:57.259107 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259008    4553 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:28:57.259144 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259130    4553 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259144    4553 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259160    4553 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259166    4553 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259171    4553 policy_none.go:50] "Start"
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259179    4553 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:28:57.259198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259187    4553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:28:57.259357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259305    4553 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:28:57.259357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259312    4553 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:28:57.259357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.259317    4553 policy_none.go:44] "Start"
Oct 31 00:28:57.263252 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.263226    4553 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:28:57.266989 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.266977    4553 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:28:57.267505 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.267485    4553 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:28:57.267505 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.267503    4553 server.go:87] "Starting device plugin registration server"
Oct 31 00:28:57.267778 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.267764    4553 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:28:57.267856 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.267781    4553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:28:57.268127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.268007    4553 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:28:57.268127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.268101    4553 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:28:57.268248 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.268138    4553 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:28:57.268248 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.268145    4553 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:28:57.268354 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.268198    4553 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:57.268127973Z","UUID":"16f94fb1-7148-46c1-ba4a-966157c0cc9b","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:28:57.272253 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.270586    4553 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:28:57.272253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.271799    4553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:28:57.272253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.271824    4553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:28:57.294183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294127    4553 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"]
Oct 31 00:28:57.294841 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294824    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.294909 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294874    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerStarted","Data":"1a348f524960f8d758687fbd20017ac281bfc5740d1f4dd365d8df07cabd2961"}
Oct 31 00:28:57.294954 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294937    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerDied","Data":"8e6a1946ab4519ad5233c9072ed090c769e3867187471f0dbba786f4e6e79c68"}
Oct 31 00:28:57.294987 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294958    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"f9318895ca5d45d45cb102c90334bd92","Type":"ContainerStarted","Data":"0130737f0495a7cb01d6059da720bb3ac440b1f5bfb83bfb4d8f7ac149bb0ce8"}
Oct 31 00:28:57.294987 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294896    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.295076 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.294975    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"2b7e31f843ae335a5983ffff48ea08a5","Type":"ContainerStarted","Data":"d0aad7d9dcdf0ef3f5d47f018fea9aa9d5f32a8d42d54627640c1419f40c7c81"}
Oct 31 00:28:57.295109 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.295096    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal" event={"ID":"2b7e31f843ae335a5983ffff48ea08a5","Type":"ContainerStarted","Data":"14536b1b3fe926c559b84130584107d094440d1f04047356b7f6f953c1762478"}
Oct 31 00:28:57.308866 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.308850    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:57.308930 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.308895    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.310639 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.310625    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:57.310727 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:57.310660    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.368025 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.368001    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:57.369267 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.369253    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:57.369326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.369281    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:57.369326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.369290    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-193.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:57.369385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.369330    4553 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.376165 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.376118    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.376165 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.376153    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.376264 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.376179    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.389624 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.389603    4553 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.389733 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.389649    4553 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.476919 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.476892    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.476919 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.476911    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.477047 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.476935    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.477047 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.476964    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.477047 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.476995    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b7e31f843ae335a5983ffff48ea08a5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"2b7e31f843ae335a5983ffff48ea08a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:57.477047 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:57.477040    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9318895ca5d45d45cb102c90334bd92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" (UID: \"f9318895ca5d45d45cb102c90334bd92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:58.149407 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.149374    4553 apiserver.go:52] "Watching apiserver"
Oct 31 00:28:58.179432 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.179412    4553 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:28:58.179915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.179894    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ck7z7","openshift-service-ca/service-ca-5fffc97bdf-9c8q4","openshift-ingress-canary/ingress-canary-jrkck","openshift-network-diagnostics/network-check-target-c6kkm","kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal","openshift-dns/dns-default-76d7d","openshift-multus/network-metrics-daemon-lhnbd","openshift-multus/multus-9bm4l","kube-system/global-pull-secret-syncer-2pb9q","kube-system/konnectivity-agent-42vpj","openshift-multus/multus-additional-cni-plugins-nmcll","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp","openshift-image-registry/node-ca-jjgcf","openshift-ovn-kubernetes/ovnkube-node-rmgdp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal","openshift-insights/periodic-gathering-5t5nf-vnx6r","openshift-network-operator/iptables-alerter-nlqbt","openshift-cluster-node-tuning-operator/tuned-7k48d","openshift-insights/insights-runtime-extractor-gnddf","openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"]
Oct 31 00:28:58.180354 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.180337    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.187297 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.187269    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.187479 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.187456    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.187853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.187832    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:28:58.188295 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.188274    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:28:58.188385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.188313    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:28:58.188811 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.188792    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.188908 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.188869    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:28:58.188961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.188946    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:28:58.189195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.189174    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:28:58.189263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.189232    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.189503 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.189484    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:28:58.189503 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.189477    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.189732 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.189715    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.190417 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.190395    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.190527 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.190493    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:28:58.190639 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.190620    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.190852 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.190833    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:28:58.190912 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.190899    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bqfb\""
Oct 31 00:28:58.191243 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.191221    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8g45\""
Oct 31 00:28:58.191392 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.191315    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.191486 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.191395    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.191583 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.191484    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.192160 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192140    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9bm5h\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192167    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b2pkn\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192173    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dnwdb\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192229    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192239    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192242    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192257    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.192263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192144    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192272    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192287    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzwtm\""
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192278    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cnz4r\""
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192175    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hxhg5\""
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192288    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.192598 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.192357    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qx27s\""
Oct 31 00:28:58.193736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.193716    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.236230 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236204    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.236326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236314    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.236326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236320    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:28:58.236436 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236396    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:28:58.236436 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236400    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.236539 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236455    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.236539 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236482    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.236539 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236491    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:28:58.236539 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236509    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.236539 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236522    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.236782 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236554    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:28:58.236782 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236643    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:28:58.236782 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236755    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:58.236915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.236787    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.237139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.237123    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:28:58.239023 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.239007    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:58.239228 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.239217    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:58.241000 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.240984    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:28:58.251012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.250998    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:58.251083 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.251035    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:58.252627 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.252614    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:58.252671 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.252642    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:28:58.263654 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.263640    4553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:28:58.280770 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280752    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.280869 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280779    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:58.280869 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280803    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwsw\" (UniqueName: \"kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:58.280869 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280819    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.280869 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280836    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.281053 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280945    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.281053 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.280988    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281053 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281024    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.281143 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281067    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.281143 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281093    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.281143 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281136    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6nb\" (UniqueName: \"kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.281236 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281187    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281236 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281218    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281312 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281245    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281312 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281274    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.281387 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281313    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.281418 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281385    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.281471 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281454    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksbs\" (UniqueName: \"kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.281518 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281478    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48764144-837f-4ae6-ac64-982b8c04c362-signing-cabundle\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.281567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281523    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cni-binary-copy\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281551    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.281567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281550    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.281721 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281610    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.281721 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281652    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47sq\" (UniqueName: \"kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.281815 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281715    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281815 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281759    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281815 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281796    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-tmp\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.281905 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281798    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.281905 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281857    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281905 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281888    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281993 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281918    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.281993 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.281965    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.282051 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282019    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.282080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282062    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.282141 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282125    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.282141 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282128    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-daemon-config\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.282204 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282135    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf22e752-9631-48b6-94da-3e7c458d7601-data-volume\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.282204 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282160    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.282204 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282184    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.282286 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282277    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.282313 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282297    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.282342 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282321    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75gr\" (UniqueName: \"kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.282407 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282340    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9rk\" (UniqueName: \"kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.282407 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282385    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282407 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282401    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282419    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.282499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282462    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbfee234-053b-4291-aecd-5c197dded041-original-pull-secret\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.282499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282461    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.282499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282494    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282515    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282539    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.282636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282540    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ba2243-f40b-4f95-a955-bd82c885d457-serviceca\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.282636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282554    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.282636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282593    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwngb\" (UniqueName: \"kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.282810 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.282810 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282750    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-env-overrides\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282816    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.282865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282850    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.282925 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282870    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.282959 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282926    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.282959 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.282944    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d383848-ee88-4158-b9f8-8dd88fdab4c4-tmp-dir\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.283021 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283001    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.283050 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283036    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d383848-ee88-4158-b9f8-8dd88fdab4c4-config-volume\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.283050 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283041    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:58.283103 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283061    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd577\" (UniqueName: \"kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577\") pod \"migrator-66df5d496b-429vc\" (UID: \"f4c055f3-44ac-423d-9c0c-85383b5d8d94\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:58.283103 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283083    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.283218 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283202    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.283253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283233    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdf6\" (UniqueName: \"kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.283294 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283254    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.283341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283298    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.283385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283350    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.283483 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283464    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.283532 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283502    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.283583 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283547    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.283583 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283579    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.283705 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283584    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-iptables-alerter-script\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.283705 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283663    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.283813 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283741    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.283813 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283774    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.283813 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283799    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.283923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.283923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283867    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovn-node-metrics-cert\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.283923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283907    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.284017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283941    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.284017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.283967    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.284017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284007    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.284114 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284039    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.284114 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284066    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.284114 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284094    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswmc\" (UniqueName: \"kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.284114 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284108    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.284222 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284069    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-tuned\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.284222 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284120    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.284222 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284188    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.284330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284220    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.284330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284261    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.284452 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284415    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtjg\" (UniqueName: \"kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.284524 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284453    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.284524 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284481    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e47f56-d809-4b6a-861b-98b7bd8241fb-tmp-dir\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.284617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284520    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.284617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284525    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.284617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284570    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.284786 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284610    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.284786 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284646    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:58.284786 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284707    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.284786 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284734    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.284786 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.284763    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:58.285208 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285179    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.285326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285293    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.285496 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285467    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-konnectivity-ca\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:58.285603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285580    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.285671 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285648    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdf44\" (UniqueName: \"kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.285739 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285699    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.285739 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285732    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.285833 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285735    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.285833 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285761    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.285935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285849    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd56\" (UniqueName: \"kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.285935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285849    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-config\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.285935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.285906    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.286073 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.286011    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.286144 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.286054    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.286190 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.286165    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjrh\" (UniqueName: \"kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.286258 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.286244    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.286465 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.286448    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e79d059-a2d5-419e-bb3a-de74bb0082e5-ovnkube-script-lib\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.296850 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.296828    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwsw\" (UniqueName: \"kubernetes.io/projected/c3e796e1-0c4b-455c-906d-98ee4c141fe0-kube-api-access-gdwsw\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:58.298040 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.298025    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75gr\" (UniqueName: \"kubernetes.io/projected/c6ad6861-24a8-40ff-8a4e-ee332a213b54-kube-api-access-h75gr\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.303499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.303461    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:58.303789 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.303771    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksbs\" (UniqueName: \"kubernetes.io/projected/22e47f56-d809-4b6a-861b-98b7bd8241fb-kube-api-access-kksbs\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.305370 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305345    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswmc\" (UniqueName: \"kubernetes.io/projected/0e79d059-a2d5-419e-bb3a-de74bb0082e5-kube-api-access-sswmc\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.305438 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305378    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:28:58.305438 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305415    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:28:58.305560 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305545    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:28:58.305774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305760    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:28:58.305816 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.305777    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:28:58.306346 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306328    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:28:58.306423 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306344    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56d49\""
Oct 31 00:28:58.306423 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306352    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wfrwt\""
Oct 31 00:28:58.306423 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306342    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4hxn\""
Oct 31 00:28:58.306423 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306338    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rmr2b\""
Oct 31 00:28:58.306626 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306364    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r62gb\""
Oct 31 00:28:58.306799 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306781    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8j7n8\""
Oct 31 00:28:58.306799 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306781    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7wwd7\""
Oct 31 00:28:58.306929 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306781    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nchkl\""
Oct 31 00:28:58.306929 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.306830    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pmz5p\""
Oct 31 00:28:58.307053 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.307034    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"periodic-gathering-5t5nf-vnx6r\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") " pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:28:58.307176 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.307162    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9rk\" (UniqueName: \"kubernetes.io/projected/bf22e752-9631-48b6-94da-3e7c458d7601-kube-api-access-fc9rk\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.309010 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.308995    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjrh\" (UniqueName: \"kubernetes.io/projected/a862323a-20e5-41f7-86bf-79f2964d1ec5-kube-api-access-dgjrh\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.309050 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.309014    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd577\" (UniqueName: \"kubernetes.io/projected/f4c055f3-44ac-423d-9c0c-85383b5d8d94-kube-api-access-kd577\") pod \"migrator-66df5d496b-429vc\" (UID: \"f4c055f3-44ac-423d-9c0c-85383b5d8d94\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-429vc"
Oct 31 00:28:58.310813 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.310798    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdf44\" (UniqueName: \"kubernetes.io/projected/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-kube-api-access-jdf44\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.310856 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.310834    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47sq\" (UniqueName: \"kubernetes.io/projected/5d383848-ee88-4158-b9f8-8dd88fdab4c4-kube-api-access-h47sq\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.311170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.311157    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdf6\" (UniqueName: \"kubernetes.io/projected/48764144-837f-4ae6-ac64-982b8c04c362-kube-api-access-mhdf6\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.311311 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.311299    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25f0fe6f-7a3d-4966-95be-31c2ee3941a6-agent-certs\") pod \"konnectivity-agent-42vpj\" (UID: \"25f0fe6f-7a3d-4966-95be-31c2ee3941a6\") " pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:28:58.313816 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.313796    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48764144-837f-4ae6-ac64-982b8c04c362-signing-key\") pod \"service-ca-5fffc97bdf-9c8q4\" (UID: \"48764144-837f-4ae6-ac64-982b8c04c362\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9c8q4"
Oct 31 00:28:58.314096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.314077    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwngb\" (UniqueName: \"kubernetes.io/projected/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kube-api-access-qwngb\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.314359 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.314341    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd56\" (UniqueName: \"kubernetes.io/projected/c61a812b-5c90-419b-bf1d-6c2cad75af52-kube-api-access-bkd56\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.314413 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.314364    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44ts\" (UniqueName: \"kubernetes.io/projected/3625a6a6-66db-49b1-8fb6-fae35972b557-kube-api-access-h44ts\") pod \"network-check-target-c6kkm\" (UID: \"3625a6a6-66db-49b1-8fb6-fae35972b557\") " pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:58.315054 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.315036    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3e796e1-0c4b-455c-906d-98ee4c141fe0-cert\") pod \"ingress-canary-jrkck\" (UID: \"c3e796e1-0c4b-455c-906d-98ee4c141fe0\") " pod="openshift-ingress-canary/ingress-canary-jrkck"
Oct 31 00:28:58.315355 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.315341    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6nb\" (UniqueName: \"kubernetes.io/projected/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-kube-api-access-hc6nb\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.319298 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.319278    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtjg\" (UniqueName: \"kubernetes.io/projected/e5ba2243-f40b-4f95-a955-bd82c885d457-kube-api-access-8wtjg\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.356844 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.356814    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jrkck_c3e796e1-0c4b-455c-906d-98ee4c141fe0/serve-healthcheck-canary/0.log"
Oct 31 00:28:58.386667 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386644    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386670    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386712    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386749    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386756    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386775    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.386795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386786    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-dbus\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386794    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-var-lib-kubelet\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386799    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386794    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-k8s-cni-cncf-io\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386814    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-etc-kubernetes\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386849    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysconfig\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386891    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386918    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386942    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386964    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386966    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386989    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bed308ea-c7aa-4772-ab2e-458e6ef7a33d-host-slash\") pod \"iptables-alerter-nlqbt\" (UID: \"bed308ea-c7aa-4772-ab2e-458e6ef7a33d\") " pod="openshift-network-operator/iptables-alerter-nlqbt"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386989    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387011    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.386968    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-netns\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387026    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387029    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-cnibin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387055    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387058    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-sysctl-conf\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387064    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-systemd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387073    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387096    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387120    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387133    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387144    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387160    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-socket-dir-parent\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387169    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387176    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-conf-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387194    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387202    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-ovn\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387215    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387235    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387243    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-bin\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387257    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387263    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-cni-multus\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387265    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-lib-modules\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387279    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387298    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-system-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387300    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387322    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-multus-certs\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387323    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387284    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-systemd\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387324    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-multus-cni-dir\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387352    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387370    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387385    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387407    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387438    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-slash\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387448    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbfee234-053b-4291-aecd-5c197dded041-kubelet-config\") pod \"global-pull-secret-syncer-2pb9q\" (UID: \"cbfee234-053b-4291-aecd-5c197dded041\") " pod="kube-system/global-pull-secret-syncer-2pb9q"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387451    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-host\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387471    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387491    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387504    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387518    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387537    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-sys\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387539    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387564    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-node-log\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387563    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387581    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-modprobe-d\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387592    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-system-cni-dir\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387594    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387612    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387623    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387648    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387648    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387708    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-netd\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387721    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e47f56-d809-4b6a-861b-98b7bd8241fb-hosts-file\") pod \"node-resolver-ck7z7\" (UID: \"22e47f56-d809-4b6a-861b-98b7bd8241fb\") " pod="openshift-dns/node-resolver-ck7z7"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387727    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cd8b314-8c3e-45a3-9395-bd88d51d1b08-cnibin\") pod \"multus-additional-cni-plugins-nmcll\" (UID: \"9cd8b314-8c3e-45a3-9395-bd88d51d1b08\") " pod="openshift-multus/multus-additional-cni-plugins-nmcll"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387724    4553 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387733    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387781    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387803    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387808    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf22e752-9631-48b6-94da-3e7c458d7601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387822    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ba2243-f40b-4f95-a955-bd82c885d457-host\") pod \"node-ca-jjgcf\" (UID: \"e5ba2243-f40b-4f95-a955-bd82c885d457\") " pod="openshift-image-registry/node-ca-jjgcf"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387829    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387839    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-run-netns\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387865    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387868    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-etc-kubernetes\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387868    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-kubelet\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387902    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.387895 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387940    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-host-var-lib-kubelet\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387944    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387978    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-etc-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.387996    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388018    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388040    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388044    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-host-cni-bin\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388051    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-device-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388061    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388086    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388095    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-systemd-units\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388096    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61a812b-5c90-419b-bf1d-6c2cad75af52-run\") pod \"tuned-7k48d\" (UID: \"c61a812b-5c90-419b-bf1d-6c2cad75af52\") " pod="openshift-cluster-node-tuning-operator/tuned-7k48d"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388108    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388132    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388148    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388154    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388160    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388134    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2aa170b9-6ca8-45d3-b0a0-7394bf27fff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9rp\" (UID: \"2aa170b9-6ca8-45d3-b0a0-7394bf27fff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9rp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388176    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388201    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6ad6861-24a8-40ff-8a4e-ee332a213b54-hostroot\") pod \"multus-9bm4l\" (UID: \"c6ad6861-24a8-40ff-8a4e-ee332a213b54\") " pod="openshift-multus/multus-9bm4l"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388199    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388224    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388246    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.388252    4553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388276    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-run-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.388324    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:58.888304663 +0000 UTC m=+1.797307310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388372    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf22e752-9631-48b6-94da-3e7c458d7601-crio-socket\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388381    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-log-socket\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.389781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.388415    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e79d059-a2d5-419e-bb3a-de74bb0082e5-var-lib-openvswitch\") pod \"ovnkube-node-rmgdp\" (UID: \"0e79d059-a2d5-419e-bb3a-de74bb0082e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:58.390542 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.389992    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a862323a-20e5-41f7-86bf-79f2964d1ec5-metrics-certs\") pod \"network-metrics-daemon-lhnbd\" (UID: \"a862323a-20e5-41f7-86bf-79f2964d1ec5\") " pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.390542 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.390016    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf22e752-9631-48b6-94da-3e7c458d7601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnddf\" (UID: \"bf22e752-9631-48b6-94da-3e7c458d7601\") " pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.481208 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.481178    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhnbd"
Oct 31 00:28:58.493996 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.493975    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnddf"
Oct 31 00:28:58.757236 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.757166    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-429vc_f4c055f3-44ac-423d-9c0c-85383b5d8d94/migrator/0.log"
Oct 31 00:28:58.787856 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.787824    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhnbd"]
Oct 31 00:28:58.791187 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:58.791156    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda862323a_20e5_41f7_86bf_79f2964d1ec5.slice/crio-d1b1516e222c55b8044bd872e4c0a15cac9b4d60eaf1845f4abf283e62b60343 WatchSource:0}: Error finding container d1b1516e222c55b8044bd872e4c0a15cac9b4d60eaf1845f4abf283e62b60343: Status 404 returned error can't find the container with id d1b1516e222c55b8044bd872e4c0a15cac9b4d60eaf1845f4abf283e62b60343
Oct 31 00:28:58.793194 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.793177    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:28:58.812289 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.812268    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnddf"]
Oct 31 00:28:58.814930 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:28:58.814905    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf22e752_9631_48b6_94da_3e7c458d7601.slice/crio-202f41537b6726f8d11c0e77cf1adae8acb83557a94750fd120031eb998e0e74 WatchSource:0}: Error finding container 202f41537b6726f8d11c0e77cf1adae8acb83557a94750fd120031eb998e0e74: Status 404 returned error can't find the container with id 202f41537b6726f8d11c0e77cf1adae8acb83557a94750fd120031eb998e0e74
Oct 31 00:28:58.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.892393    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:58.892531 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.892518    4553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:58.892577 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:58.892569    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:59.892555913 +0000 UTC m=+2.801558546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:58.961192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:58.961168    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:28:59.246463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.246420    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnddf" event={"ID":"bf22e752-9631-48b6-94da-3e7c458d7601","Type":"ContainerStarted","Data":"9b81b6215adc9841cc10e5dcdb3b579faa56f6d23bef8ce6a86351b9ed37435c"}
Oct 31 00:28:59.246959 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.246510    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnddf" event={"ID":"bf22e752-9631-48b6-94da-3e7c458d7601","Type":"ContainerStarted","Data":"202f41537b6726f8d11c0e77cf1adae8acb83557a94750fd120031eb998e0e74"}
Oct 31 00:28:59.247546 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.247519    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhnbd" event={"ID":"a862323a-20e5-41f7-86bf-79f2964d1ec5","Type":"ContainerStarted","Data":"d1b1516e222c55b8044bd872e4c0a15cac9b4d60eaf1845f4abf283e62b60343"}
Oct 31 00:28:59.494175 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.494141    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:59.697023 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.696974    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:59.869052 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.869014    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:28:59.899421 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.899384    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:59.900244 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:28:59.900223    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:28:59.900371 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:59.900357    4553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:59.900432 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:28:59.900422    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:01.900405346 +0000 UTC m=+4.809407999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:29:00.087392 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:00.087312    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:00.291784 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:00.291730    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:00.438652 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:00.438630    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:29:00.438882 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:00.438866    4553 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:01.256294 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.256259    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnddf" event={"ID":"bf22e752-9631-48b6-94da-3e7c458d7601","Type":"ContainerStarted","Data":"b2a82ae054804c4801c309512756bc212fa1bf51655fa26246d91ea2574971ee"}
Oct 31 00:29:01.258094 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.258065    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhnbd" event={"ID":"a862323a-20e5-41f7-86bf-79f2964d1ec5","Type":"ContainerStarted","Data":"98ace896157a09768387da271ec8d3fd93b2a6fc95d767a0938356b11770e0a7"}
Oct 31 00:29:01.258206 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.258115    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhnbd" event={"ID":"a862323a-20e5-41f7-86bf-79f2964d1ec5","Type":"ContainerStarted","Data":"167f8338f021b20d983b74377fc25ef47d293751e0a77e3f6dd3bbd5cac9acdf"}
Oct 31 00:29:01.485485 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.485446    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:01.708966 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.708915    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:01.888397 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.888350    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:01.913449 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:01.913409    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:01.913608 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:29:01.913544    4553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:29:01.913666 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:29:01.913623    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls podName:5d383848-ee88-4158-b9f8-8dd88fdab4c4 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:05.913598416 +0000 UTC m=+8.822601053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls") pod "dns-default-76d7d" (UID: "5d383848-ee88-4158-b9f8-8dd88fdab4c4") : secret "dns-default-metrics-tls" not found
Oct 31 00:29:02.091194 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:02.091109    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:02.238475 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:02.238436    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:02.265698 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:02.265643    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnddf" event={"ID":"bf22e752-9631-48b6-94da-3e7c458d7601","Type":"ContainerStarted","Data":"2d816508126153719416f10f816cc86409c557aa9389c042c3be288d63fd8d86"}
Oct 31 00:29:02.377515 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:02.377435    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:02.522050 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:02.522019    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:03.495712 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:03.495650    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:03.692435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:03.692399    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:03.847875 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:03.847800    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:03.988353 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:03.988319    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:04.145763 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:04.145661    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:04.291711 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:04.291650    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:04.434219 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:04.434191    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.240342 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.240306    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.387643 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.387602    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.534507 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.534434    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.698743 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.698708    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.881632 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.881555    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:05.940328 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.940302    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:05.942797 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.942769    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d383848-ee88-4158-b9f8-8dd88fdab4c4-metrics-tls\") pod \"dns-default-76d7d\" (UID: \"5d383848-ee88-4158-b9f8-8dd88fdab4c4\") " pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:05.993502 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:05.993480    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:06.093077 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.093044    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:06.267459 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.267355    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76d7d"]
Oct 31 00:29:06.270158 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:06.270137    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d383848_ee88_4158_b9f8_8dd88fdab4c4.slice/crio-64401a651b265a61feddc647378597fc73942bc7ab387e44e891a13f76e91294 WatchSource:0}: Error finding container 64401a651b265a61feddc647378597fc73942bc7ab387e44e891a13f76e91294: Status 404 returned error can't find the container with id 64401a651b265a61feddc647378597fc73942bc7ab387e44e891a13f76e91294
Oct 31 00:29:06.278932 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.278910    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76d7d" event={"ID":"5d383848-ee88-4158-b9f8-8dd88fdab4c4","Type":"ContainerStarted","Data":"64401a651b265a61feddc647378597fc73942bc7ab387e44e891a13f76e91294"}
Oct 31 00:29:06.299616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.299596    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-42vpj"
Oct 31 00:29:06.303777 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.303751    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:06.887535 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.887501    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:06.965379 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:06.965337    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmgdp"
Oct 31 00:29:07.088459 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:07.088421    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:07.305441 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:07.305404    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:07.454974 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:07.454926    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:07.621028 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:07.620925    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:07.760463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:07.760434    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:08.287814 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:08.287782    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76d7d" event={"ID":"5d383848-ee88-4158-b9f8-8dd88fdab4c4","Type":"ContainerStarted","Data":"2f32c3467daf9d66af4be818ff1e6a61e9845994f7b55d5be667e0edb9e4bfc4"}
Oct 31 00:29:08.287989 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:08.287837    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76d7d" event={"ID":"5d383848-ee88-4158-b9f8-8dd88fdab4c4","Type":"ContainerStarted","Data":"777bdff6ba696477caaa350c371dff6db342aca54ea420cfa283953b557c0800"}
Oct 31 00:29:08.287989 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:08.287933    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:09.661753 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.661719    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:09.872542 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.872506    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:09.923004 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.922941    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"]
Oct 31 00:29:09.926000 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.925982    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:09.929793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.929773    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-l7s2h"]
Oct 31 00:29:09.932601 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.932583    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:09.937716 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.937666    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"]
Oct 31 00:29:09.939175 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.939143    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qmgns\""
Oct 31 00:29:09.939308 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.939233    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 00:29:09.950468 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.950446    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l7s2h"]
Oct 31 00:29:09.950699 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.950665    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 00:29:09.950777 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.950724    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 00:29:09.967994 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:09.967966    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b91d7c0-7891-4c5c-af83-100228221cb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b25mx\" (UID: \"7b91d7c0-7891-4c5c-af83-100228221cb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:10.012502 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.012478    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b97446cdb-xckwp"]
Oct 31 00:29:10.016965 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.016947    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.032037 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.032016    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b97446cdb-xckwp"]
Oct 31 00:29:10.038200 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038170    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:29:10.038597 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038575    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:29:10.038817 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038804    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-42gw8\""
Oct 31 00:29:10.038863 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038827    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xbmsm\""
Oct 31 00:29:10.038863 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038841    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:29:10.038923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.038873    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:29:10.068301 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068275    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqfn\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-kube-api-access-4hqfn\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068317    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9cz\" (UniqueName: \"kubernetes.io/projected/91ee8dff-fe5b-4c23-a771-ea7282e170f1-kube-api-access-fv9cz\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.068445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068343    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ee8dff-fe5b-4c23-a771-ea7282e170f1-tmp\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.068445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068389    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:10.068445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a5d5943-6aba-4318-8000-62ed7ecf1a39-ca-trust-extracted\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068479    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b91d7c0-7891-4c5c-af83-100228221cb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b25mx\" (UID: \"7b91d7c0-7891-4c5c-af83-100228221cb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:10.068605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068504    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-certificates\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068524    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-trusted-ca\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068581    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-image-registry-private-configuration\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068610    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-installation-pull-secrets\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068645    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-bound-sa-token\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.068747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.068689    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-tls\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.071162 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.071139    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7b91d7c0-7891-4c5c-af83-100228221cb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b25mx\" (UID: \"7b91d7c0-7891-4c5c-af83-100228221cb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:10.169457 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169423    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-certificates\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169457 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169457    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-trusted-ca\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169477    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-image-registry-private-configuration\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169614    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-installation-pull-secrets\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169663    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-bound-sa-token\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169721    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-tls\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169775    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqfn\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-kube-api-access-4hqfn\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.169933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169824    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9cz\" (UniqueName: \"kubernetes.io/projected/91ee8dff-fe5b-4c23-a771-ea7282e170f1-kube-api-access-fv9cz\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.169933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169858    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ee8dff-fe5b-4c23-a771-ea7282e170f1-tmp\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.169933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.169891    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a5d5943-6aba-4318-8000-62ed7ecf1a39-ca-trust-extracted\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.170285 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.170261    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a5d5943-6aba-4318-8000-62ed7ecf1a39-ca-trust-extracted\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.170389 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.170370    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-certificates\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.170460 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.170439    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91ee8dff-fe5b-4c23-a771-ea7282e170f1-tmp\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.170706 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.170446    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a5d5943-6aba-4318-8000-62ed7ecf1a39-trusted-ca\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.172214 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.172192    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-image-registry-private-configuration\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.172285 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.172272    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-registry-tls\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.172574 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.172559    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a5d5943-6aba-4318-8000-62ed7ecf1a39-installation-pull-secrets\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.185739 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.185672    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-bound-sa-token\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.186006 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.185987    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9cz\" (UniqueName: \"kubernetes.io/projected/91ee8dff-fe5b-4c23-a771-ea7282e170f1-kube-api-access-fv9cz\") pod \"downloads-6dd789c8b7-l7s2h\" (UID: \"91ee8dff-fe5b-4c23-a771-ea7282e170f1\") " pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.186213 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.186196    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqfn\" (UniqueName: \"kubernetes.io/projected/7a5d5943-6aba-4318-8000-62ed7ecf1a39-kube-api-access-4hqfn\") pod \"image-registry-5b97446cdb-xckwp\" (UID: \"7a5d5943-6aba-4318-8000-62ed7ecf1a39\") " pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.209369 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.209344    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:10.238416 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.238388    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:10.245736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.245717    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:10.327044 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.327018    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:10.517454 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.517358    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"]
Oct 31 00:29:10.519792 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:10.519765    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91d7c0_7891_4c5c_af83_100228221cb1.slice/crio-e4d4bdec30989998007c2442a6ffc2ceae99766a3f91b6d37bacaa3aa24acc1c WatchSource:0}: Error finding container e4d4bdec30989998007c2442a6ffc2ceae99766a3f91b6d37bacaa3aa24acc1c: Status 404 returned error can't find the container with id e4d4bdec30989998007c2442a6ffc2ceae99766a3f91b6d37bacaa3aa24acc1c
Oct 31 00:29:10.529080 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.529052    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l7s2h"]
Oct 31 00:29:10.530963 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:10.530942    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ee8dff_fe5b_4c23_a771_ea7282e170f1.slice/crio-623f5a28e5f652245ddd1570bec82cce94b9b0c6ac91450d211678ca469d3c60 WatchSource:0}: Error finding container 623f5a28e5f652245ddd1570bec82cce94b9b0c6ac91450d211678ca469d3c60: Status 404 returned error can't find the container with id 623f5a28e5f652245ddd1570bec82cce94b9b0c6ac91450d211678ca469d3c60
Oct 31 00:29:10.604531 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:10.604467    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b97446cdb-xckwp"]
Oct 31 00:29:10.606739 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:10.606716    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5d5943_6aba_4318_8000_62ed7ecf1a39.slice/crio-219c1a76efbcf89680fc1c3aea0306ec6d19f564552da9c958a2d2b233257b81 WatchSource:0}: Error finding container 219c1a76efbcf89680fc1c3aea0306ec6d19f564552da9c958a2d2b233257b81: Status 404 returned error can't find the container with id 219c1a76efbcf89680fc1c3aea0306ec6d19f564552da9c958a2d2b233257b81
Oct 31 00:29:11.304627 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.304588    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l7s2h" event={"ID":"91ee8dff-fe5b-4c23-a771-ea7282e170f1","Type":"ContainerStarted","Data":"623f5a28e5f652245ddd1570bec82cce94b9b0c6ac91450d211678ca469d3c60"}
Oct 31 00:29:11.307050 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.307023    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp" event={"ID":"7a5d5943-6aba-4318-8000-62ed7ecf1a39","Type":"ContainerStarted","Data":"a501ffb9b5ead5a1ac7a8f454cc9ad84e73ff8fc7986b41d87896f180e3ea629"}
Oct 31 00:29:11.307183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.307064    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp" event={"ID":"7a5d5943-6aba-4318-8000-62ed7ecf1a39","Type":"ContainerStarted","Data":"219c1a76efbcf89680fc1c3aea0306ec6d19f564552da9c958a2d2b233257b81"}
Oct 31 00:29:11.307183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.307111    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:11.308541 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.308515    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx" event={"ID":"7b91d7c0-7891-4c5c-af83-100228221cb1","Type":"ContainerStarted","Data":"e4d4bdec30989998007c2442a6ffc2ceae99766a3f91b6d37bacaa3aa24acc1c"}
Oct 31 00:29:11.336016 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:11.335963    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp" podStartSLOduration=2.335947593 podStartE2EDuration="2.335947593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:11.335289179 +0000 UTC m=+14.244291847" watchObservedRunningTime="2026-10-31 00:29:11.335947593 +0000 UTC m=+14.244950248"
Oct 31 00:29:12.316770 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:12.316732    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx" event={"ID":"7b91d7c0-7891-4c5c-af83-100228221cb1","Type":"ContainerStarted","Data":"53b713d7a4bc3955a2113708309a34e49ae9bd00e272d524f7e1e8f592875b53"}
Oct 31 00:29:13.029830 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.029779    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx" podStartSLOduration=2.87226736 podStartE2EDuration="4.029761956s" totalImagesPullingTime="1.157494596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:12.339364274 +0000 UTC m=+15.248366928" watchObservedRunningTime="2026-10-31 00:29:13.029761956 +0000 UTC m=+15.938764609"
Oct 31 00:29:13.030227 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.030211    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:13.033446 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.033422    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.040286 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040194    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 00:29:13.040406 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040342    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wfp74\""
Oct 31 00:29:13.040406 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040371    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 00:29:13.040406 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040386    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 00:29:13.040406 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040344    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 00:29:13.040632 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.040423    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 00:29:13.045121 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.045063    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:13.076921 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.076889    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:13.094406 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094381    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094530 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094434    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094530 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094478    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094638 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094541    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094646    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fv5\" (UniqueName: \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094707    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.094822 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.094754    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195398 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195362    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fv5\" (UniqueName: \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195417    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195465    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195518    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195555    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195829 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195580    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.195829 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195615    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.196003 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.195986    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.196326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.196304    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.196391 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.196353    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.196391 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.196364    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.198417 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.198394    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.198700 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.198664    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.210513 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.210492    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fv5\" (UniqueName: \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\") pod \"console-677d75b9b4-hlrx5\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") " pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.321323 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.321090    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:13.325963 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.325938    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b25mx"
Oct 31 00:29:13.346316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.346130    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:13.633884 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:13.633808    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:13.638126 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:13.638097    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369a4809_d3f3_4057_8382_8d4db94db914.slice/crio-9f5e91af836ca07f196615ad3b09cc57427aeb3906e2b740c6c47457c87a745a WatchSource:0}: Error finding container 9f5e91af836ca07f196615ad3b09cc57427aeb3906e2b740c6c47457c87a745a: Status 404 returned error can't find the container with id 9f5e91af836ca07f196615ad3b09cc57427aeb3906e2b740c6c47457c87a745a
Oct 31 00:29:14.333455 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.333424    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677d75b9b4-hlrx5" event={"ID":"369a4809-d3f3-4057-8382-8d4db94db914","Type":"ContainerStarted","Data":"9f5e91af836ca07f196615ad3b09cc57427aeb3906e2b740c6c47457c87a745a"}
Oct 31 00:29:14.442586 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.442546    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:14.447617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.447557    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"]
Oct 31 00:29:14.451708 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.451370    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.459203 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.459177    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:29:14.459579 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.459562    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:29:14.459729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.459702    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:29:14.459792 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.459768    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 00:29:14.461093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.460986    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 00:29:14.461093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.461066    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nph22\""
Oct 31 00:29:14.462025 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.462005    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"]
Oct 31 00:29:14.519130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.519093    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.519287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.519169    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/854ab973-d307-4302-a733-ac0aae740462-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.519287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.519218    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdqt\" (UniqueName: \"kubernetes.io/projected/854ab973-d307-4302-a733-ac0aae740462-kube-api-access-txdqt\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.519416 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.519308    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.582253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.582224    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:14.620340 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.620271    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/854ab973-d307-4302-a733-ac0aae740462-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.620340 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.620314    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdqt\" (UniqueName: \"kubernetes.io/projected/854ab973-d307-4302-a733-ac0aae740462-kube-api-access-txdqt\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.620525 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.620371    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.620525 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.620409    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.621778 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.621745    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/854ab973-d307-4302-a733-ac0aae740462-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.623580 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.623560    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.623913 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.623886    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/854ab973-d307-4302-a733-ac0aae740462-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.636484 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.636463    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdqt\" (UniqueName: \"kubernetes.io/projected/854ab973-d307-4302-a733-ac0aae740462-kube-api-access-txdqt\") pod \"prometheus-operator-9cd89b74c-m9wb8\" (UID: \"854ab973-d307-4302-a733-ac0aae740462\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:14.770234 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:14.770192    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"
Oct 31 00:29:15.060644 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:15.060607    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8"]
Oct 31 00:29:15.063722 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:15.063689    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854ab973_d307_4302_a733_ac0aae740462.slice/crio-93e9785a71fa0f03c5a4c8c31b38d8bc2d88a5d2113286481428a045be438df4 WatchSource:0}: Error finding container 93e9785a71fa0f03c5a4c8c31b38d8bc2d88a5d2113286481428a045be438df4: Status 404 returned error can't find the container with id 93e9785a71fa0f03c5a4c8c31b38d8bc2d88a5d2113286481428a045be438df4
Oct 31 00:29:15.339655 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:15.339576    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8" event={"ID":"854ab973-d307-4302-a733-ac0aae740462","Type":"ContainerStarted","Data":"93e9785a71fa0f03c5a4c8c31b38d8bc2d88a5d2113286481428a045be438df4"}
Oct 31 00:29:16.079855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:16.079815    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:16.227853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:16.227714    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:17.033256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.033217    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:17.350732 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.350630    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677d75b9b4-hlrx5" event={"ID":"369a4809-d3f3-4057-8382-8d4db94db914","Type":"ContainerStarted","Data":"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"}
Oct 31 00:29:17.354302 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.354258    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8" event={"ID":"854ab973-d307-4302-a733-ac0aae740462","Type":"ContainerStarted","Data":"9577e6d04b8c2067cc1c335787dbdcc5339f80c84c345cc7efc09f8afda470db"}
Oct 31 00:29:17.354464 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.354327    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8" event={"ID":"854ab973-d307-4302-a733-ac0aae740462","Type":"ContainerStarted","Data":"a38f957ba90582d8ae10e3737ac1af5c9e61c80b847a25e4b24f5859e3b6090a"}
Oct 31 00:29:17.375993 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.375946    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-677d75b9b4-hlrx5" podStartSLOduration=0.871727752 podStartE2EDuration="4.375933844s" totalImagesPullingTime="3.504206092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:17.375404988 +0000 UTC m=+20.284407642" watchObservedRunningTime="2026-10-31 00:29:17.375933844 +0000 UTC m=+20.284936480"
Oct 31 00:29:17.404400 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:17.404347    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m9wb8" podStartSLOduration=1.8736268379999999 podStartE2EDuration="3.404333491s" totalImagesPullingTime="1.530706653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:17.403792368 +0000 UTC m=+20.312795021" watchObservedRunningTime="2026-10-31 00:29:17.404333491 +0000 UTC m=+20.313336144"
Oct 31 00:29:18.293612 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:18.293584    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-76d7d"
Oct 31 00:29:18.878327 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:18.878287    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:19.013004 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:19.012970    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:19.682304 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:19.682253    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:19.902931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:19.902898    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:19.981588 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:19.981515    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h5kc6"]
Oct 31 00:29:19.985316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:19.985292    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.059574 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.059532    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:20.068570 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068542    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068734 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068598    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-root\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068734 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068710    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-metrics-client-ca\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068837 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068754    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-sys\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068837 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068783    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-wtmp\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.068983 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.068961    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gmh\" (UniqueName: \"kubernetes.io/projected/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-kube-api-access-54gmh\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.069108 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.069018    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-tls\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.069108 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.069046    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-textfile\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.069453 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.069438    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:29:20.069853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.069834    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:29:20.070012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.069989    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lwv25\""
Oct 31 00:29:20.070109 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.070017    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:29:20.169568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169533    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-tls\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169571    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-textfile\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169844 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169618    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169844 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169778    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-root\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169953 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169844    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-metrics-client-ca\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169953 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169875    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-root\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169953 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169896    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-sys\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169953 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169937    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-wtmp\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.169953 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169944    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-sys\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.169966    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.170005    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gmh\" (UniqueName: \"kubernetes.io/projected/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-kube-api-access-54gmh\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.170113    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-textfile\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.170111    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-wtmp\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170529 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.170508    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-metrics-client-ca\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.170580 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.170505    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.172896 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.172870    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.173136 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.173118    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-node-exporter-tls\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.185372 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.185353    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gmh\" (UniqueName: \"kubernetes.io/projected/7e1d95e0-dcab-42de-9384-3f1ff3b506b0-kube-api-access-54gmh\") pod \"node-exporter-h5kc6\" (UID: \"7e1d95e0-dcab-42de-9384-3f1ff3b506b0\") " pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.219281 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.219239    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:20.303194 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.303122    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h5kc6"
Oct 31 00:29:20.316335 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:20.316304    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1d95e0_dcab_42de_9384_3f1ff3b506b0.slice/crio-e206dfc79601db6b3ec99ef30ebe0bce3946cda804420d7d6fffca161d42f4b9 WatchSource:0}: Error finding container e206dfc79601db6b3ec99ef30ebe0bce3946cda804420d7d6fffca161d42f4b9: Status 404 returned error can't find the container with id e206dfc79601db6b3ec99ef30ebe0bce3946cda804420d7d6fffca161d42f4b9
Oct 31 00:29:20.366754 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.366709    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h5kc6" event={"ID":"7e1d95e0-dcab-42de-9384-3f1ff3b506b0","Type":"ContainerStarted","Data":"e206dfc79601db6b3ec99ef30ebe0bce3946cda804420d7d6fffca161d42f4b9"}
Oct 31 00:29:20.371116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.371058    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:20.527198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:20.527102    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:21.000241 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:21.000201    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:21.203966 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:21.203935    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:21.411647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:21.411572    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:21.571443 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:21.571410    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:22.084750 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:22.084626    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:22.234415 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:22.234307    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:22.382294 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:22.382220    4553 generic.go:356] "Generic (PLEG): container finished" podID="7e1d95e0-dcab-42de-9384-3f1ff3b506b0" containerID="d5946961924cb06ede35837dbe203df74b6306e8ad59021ff7ae337a1e3cab57" exitCode=0
Oct 31 00:29:22.382294 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:22.382280    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h5kc6" event={"ID":"7e1d95e0-dcab-42de-9384-3f1ff3b506b0","Type":"ContainerDied","Data":"d5946961924cb06ede35837dbe203df74b6306e8ad59021ff7ae337a1e3cab57"}
Oct 31 00:29:23.019116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.019066    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:23.177877 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.177826    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:23.346978 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.346900    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:23.347202 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.347181    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:23.352745 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.352721    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:23.389285 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.389250    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h5kc6" event={"ID":"7e1d95e0-dcab-42de-9384-3f1ff3b506b0","Type":"ContainerStarted","Data":"638211a1475644cf3d8c41533c611e7c47c0dc89fab6022b667dbec5e286ff04"}
Oct 31 00:29:23.389445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.389302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h5kc6" event={"ID":"7e1d95e0-dcab-42de-9384-3f1ff3b506b0","Type":"ContainerStarted","Data":"dc76a2d373477ac388007d32d2741c805bdb65c4d8006c6f530997b1afdd9dd4"}
Oct 31 00:29:23.393407 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.393384    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:23.419642 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.419601    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h5kc6" podStartSLOduration=3.1291706169999998 podStartE2EDuration="4.419588504s" totalImagesPullingTime="1.20814273s" totalInitContainerRuntime="82.275157ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:23.418878952 +0000 UTC m=+26.327881619" watchObservedRunningTime="2026-10-31 00:29:23.419588504 +0000 UTC m=+26.328591157"
Oct 31 00:29:23.682173 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.682138    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:23.858208 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:23.858170    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:24.716442 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:24.716407    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:24.915335 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:24.915280    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:25.063347 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:25.063266    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:25.652808 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:25.652769    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:25.787334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:25.787286    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:26.360820 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:26.360715    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:27.133165 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:29:27.133089    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:29:27.457260 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:27.457228    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:27.677097 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:27.677061    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:27.812628 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:27.812555    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:28.413760 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:28.413719    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l7s2h" event={"ID":"91ee8dff-fe5b-4c23-a771-ea7282e170f1","Type":"ContainerStarted","Data":"f842828ef5da7a6e27e5c0aa093bd691af1d881b04bb1758ea67e4b6c0ef2399"}
Oct 31 00:29:28.414247 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:28.414097    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:28.429144 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:28.429119    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-l7s2h"
Oct 31 00:29:28.443247 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:28.443194    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-l7s2h" podStartSLOduration=2.111693039 podStartE2EDuration="19.443178097s" totalImagesPullingTime="17.331485058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:28.442579742 +0000 UTC m=+31.351582397" watchObservedRunningTime="2026-10-31 00:29:28.443178097 +0000 UTC m=+31.352180752"
Oct 31 00:29:29.872513 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:29.872475    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c6kkm"
Oct 31 00:29:31.209604 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:31.209571    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:31.458121 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:31.458083    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:31.630773 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:31.630671    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:32.321659 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:32.321625    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b97446cdb-xckwp"
Oct 31 00:29:32.642253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:32.642167    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:32.821852 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:32.821817    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:33.872735 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:33.872695    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:34.028703 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:34.028649    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:35.243797 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:35.243759    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:35.385399 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:35.385357    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:36.036441 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.036405    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:29:36.070197 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.070166    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:29:36.070365 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.070296    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.082914 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.082891    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 00:29:36.212361 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212322    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212528 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212389    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212528 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212510    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212550    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212577    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212630    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212800 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212671    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2v8\" (UniqueName: \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.212800 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.212773    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313165 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313090    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313165 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313145    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313189    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313220    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313247    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313270    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313300    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2v8\" (UniqueName: \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.313695 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313326    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.314017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313824    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.314017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.313962    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.314538 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.314510    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.315191 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.315144    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.315305 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.315259    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.316591 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.316568    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.318302 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.318282    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.331374 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.331353    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2v8\" (UniqueName: \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\") pod \"console-58ffb96544-q5skl\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") " pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.411769 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.411741    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:36.585910 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.585831    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:36.728216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.728186    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:29:36.731472 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:29:36.731441    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ffc8a_3df4_4e5f_9d46_6d03393ed396.slice/crio-786ffd7ab8037db7d2b3b54d47b6129c58aad497a431c19644153427062df163 WatchSource:0}: Error finding container 786ffd7ab8037db7d2b3b54d47b6129c58aad497a431c19644153427062df163: Status 404 returned error can't find the container with id 786ffd7ab8037db7d2b3b54d47b6129c58aad497a431c19644153427062df163
Oct 31 00:29:36.744734 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:36.744659    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:37.450376 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:37.450340    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ffb96544-q5skl" event={"ID":"169ffc8a-3df4-4e5f-9d46-6d03393ed396","Type":"ContainerStarted","Data":"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"}
Oct 31 00:29:37.450828 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:37.450410    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ffb96544-q5skl" event={"ID":"169ffc8a-3df4-4e5f-9d46-6d03393ed396","Type":"ContainerStarted","Data":"786ffd7ab8037db7d2b3b54d47b6129c58aad497a431c19644153427062df163"}
Oct 31 00:29:37.480520 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:37.480472    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58ffb96544-q5skl" podStartSLOduration=1.480460607 podStartE2EDuration="1.480460607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:37.479787834 +0000 UTC m=+40.388790488" watchObservedRunningTime="2026-10-31 00:29:37.480460607 +0000 UTC m=+40.389463260"
Oct 31 00:29:43.358965 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:43.358868    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:43.546172 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:43.546137    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:44.722632 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:44.722594    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:44.916780 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:44.916740    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:45.087055 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:45.086971    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:46.016690 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:46.016652    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:46.412780 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:46.412751    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:46.412935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:46.412809    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:46.417611 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:46.417590    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:46.482250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:46.482224    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:29:50.249724 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:50.249688    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:50.402508 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:50.402477    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:52.483714 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.483670    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-677d75b9b4-hlrx5" podUID="369a4809-d3f3-4057-8382-8d4db94db914" containerName="console" containerID="cri-o://874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214" gracePeriod=15
Oct 31 00:29:52.773837 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.773811    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:52.836470 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836444    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836587 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836477    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836587 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836507    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\" (UniqueName: \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836587 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836544    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836587 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836574    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\" (UniqueName: \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836618    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836664    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\") pod \"369a4809-d3f3-4057-8382-8d4db94db914\" (UID: \"369a4809-d3f3-4057-8382-8d4db94db914\") "
Oct 31 00:29:52.836936 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836908    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:52.836999 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.836918    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:52.837192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.837163    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:52.837192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.837169    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:52.838860 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.838830    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:52.838988 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.838965    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:52.839074 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.839052    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5" pod "369a4809-d3f3-4057-8382-8d4db94db914" (UID: "369a4809-d3f3-4057-8382-8d4db94db914"). InnerVolumeSpecName "kube-api-access-89fv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:52.937918 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937887    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-service-ca\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.937918 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937915    4553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.938062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937930    4553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-oauth-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.938062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937942    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fv5\" (UniqueName: \"kubernetes.io/projected/369a4809-d3f3-4057-8382-8d4db94db914-kube-api-access-89fv5\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.938062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937956    4553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369a4809-d3f3-4057-8382-8d4db94db914-console-oauth-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.938062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937968    4553 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369a4809-d3f3-4057-8382-8d4db94db914-tmp\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:52.938062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:52.937981    4553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369a4809-d3f3-4057-8382-8d4db94db914-console-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.506237 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.506164    4553 generic.go:356] "Generic (PLEG): container finished" podID="369a4809-d3f3-4057-8382-8d4db94db914" containerID="874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214" exitCode=0
Oct 31 00:29:53.506237 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.506208    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677d75b9b4-hlrx5"
Oct 31 00:29:53.506649 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.506249    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677d75b9b4-hlrx5" event={"ID":"369a4809-d3f3-4057-8382-8d4db94db914","Type":"ContainerDied","Data":"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"}
Oct 31 00:29:53.506649 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.506302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677d75b9b4-hlrx5" event={"ID":"369a4809-d3f3-4057-8382-8d4db94db914","Type":"ContainerDied","Data":"9f5e91af836ca07f196615ad3b09cc57427aeb3906e2b740c6c47457c87a745a"}
Oct 31 00:29:53.506649 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.506326    4553 scope.go:122] "RemoveContainer" containerID="874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"
Oct 31 00:29:53.514775 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.514756    4553 scope.go:122] "RemoveContainer" containerID="874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"
Oct 31 00:29:53.515067 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:29:53.515048    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214\": container with ID starting with 874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214 not found: ID does not exist" containerID="874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"
Oct 31 00:29:53.515117 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.515075    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214"} err="failed to get container status \"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214\": rpc error: code = NotFound desc = could not find container \"874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214\": container with ID starting with 874d0f68d64a5238aca890e45dadb26702cc91d188feef56955476cf53a90214 not found: ID does not exist"
Oct 31 00:29:53.528531 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.528503    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:53.531495 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:53.531478    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-677d75b9b4-hlrx5"]
Oct 31 00:29:54.080831 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:54.080792    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:54.221865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:54.221832    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:55.199086 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:55.199058    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369a4809-d3f3-4057-8382-8d4db94db914" path="/var/lib/kubelet/pods/369a4809-d3f3-4057-8382-8d4db94db914/volumes"
Oct 31 00:29:56.682337 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:56.682274    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:56.818427 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:29:56.818395    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:29:57.129720 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:29:57.129620    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:30:00.625117 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.625083    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:30:00.625512 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.625272    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" podUID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" containerName="insights-gathering" containerID="cri-o://51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15" gracePeriod=30
Oct 31 00:30:00.723638 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.723606    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2qbts-74vm8"]
Oct 31 00:30:00.723914 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.723901    4553 cpu_manager.go:414] "containerMap: removing container" podUID="369a4809-d3f3-4057-8382-8d4db94db914" containerName="console"
Oct 31 00:30:00.723964 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.723916    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369a4809-d3f3-4057-8382-8d4db94db914" containerName="console"
Oct 31 00:30:00.724000 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.723972    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="369a4809-d3f3-4057-8382-8d4db94db914" containerName="console"
Oct 31 00:30:00.757479 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.757444    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2qbts-74vm8"]
Oct 31 00:30:00.757659 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.757552    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.787717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.787651    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:00.792013 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.791981    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vcf\" (UniqueName: \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.792123 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.792055    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.792123 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.792085    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.792250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.792120    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.892755 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.892661    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.892755 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.892705    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.892755 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.892728    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.893033 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.892789    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vcf\" (UniqueName: \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.893216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.893194    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.893472 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.893449    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.893663 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.893641    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.908417 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.908392    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vcf\" (UniqueName: \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\") pod \"periodic-gathering-2qbts-74vm8\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") " pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:00.933594 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.933548    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:00.955791 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.955764    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:30:00.993374 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993339    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") pod \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") "
Oct 31 00:30:00.993553 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993408    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") pod \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") "
Oct 31 00:30:00.993553 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993446    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") pod \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") "
Oct 31 00:30:00.993553 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993490    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") pod \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\" (UID: \"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971\") "
Oct 31 00:30:00.993801 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993776    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path" pod "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" (UID: "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:00.993912 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993886    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle" pod "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" (UID: "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:00.993948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.993936    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle" pod "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" (UID: "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:00.996536 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:00.996514    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd" pod "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" (UID: "3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971"). InnerVolumeSpecName "kube-api-access-j9mjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:01.089169 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.089131    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:30:01.094475 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.094451    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-trusted-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:01.094475 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.094476    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-service-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:01.094609 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.094488    4553 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-archives-path\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:01.094609 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.094498    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9mjd\" (UniqueName: \"kubernetes.io/projected/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971-kube-api-access-j9mjd\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:01.403723 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.403495    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2qbts-74vm8"]
Oct 31 00:30:01.406535 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:30:01.406511    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb2c039_eb2d_4b2e_bdc0_5c446e609a92.slice/crio-97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb WatchSource:0}: Error finding container 97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb: Status 404 returned error can't find the container with id 97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb
Oct 31 00:30:01.536414 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.535914    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2qbts-74vm8" event={"ID":"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92","Type":"ContainerStarted","Data":"aaf4ea65657a245d9a114eb6afe500f63d16b75223f88aa9419319a7b7f5abbc"}
Oct 31 00:30:01.536414 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.535977    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2qbts-74vm8" event={"ID":"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92","Type":"ContainerStarted","Data":"97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb"}
Oct 31 00:30:01.537550 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.537522    4553 generic.go:356] "Generic (PLEG): container finished" podID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" containerID="51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15" exitCode=2
Oct 31 00:30:01.537711 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.537566    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" event={"ID":"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971","Type":"ContainerDied","Data":"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"}
Oct 31 00:30:01.537711 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.537583    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r"
Oct 31 00:30:01.537711 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.537608    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5t5nf-vnx6r" event={"ID":"3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971","Type":"ContainerDied","Data":"155349b096d8dfff8a914999d3db6545c43b468ef9242d761d736a63205a7df2"}
Oct 31 00:30:01.537711 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.537631    4553 scope.go:122] "RemoveContainer" containerID="51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"
Oct 31 00:30:01.547045 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.546996    4553 scope.go:122] "RemoveContainer" containerID="51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"
Oct 31 00:30:01.547351 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:30:01.547321    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15\": container with ID starting with 51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15 not found: ID does not exist" containerID="51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"
Oct 31 00:30:01.547417 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.547353    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15"} err="failed to get container status \"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15\": rpc error: code = NotFound desc = could not find container \"51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15\": container with ID starting with 51a064687ca4e2b19e023b3369f2b1c2bbd5fbfd8727353713b02dcb68e72e15 not found: ID does not exist"
Oct 31 00:30:01.560148 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.560097    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2qbts-74vm8" podStartSLOduration=1.5600846430000002 podStartE2EDuration="1.560084643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:01.559442028 +0000 UTC m=+64.468444683" watchObservedRunningTime="2026-10-31 00:30:01.560084643 +0000 UTC m=+64.469087296"
Oct 31 00:30:01.575538 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.575508    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:30:01.583978 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:01.583954    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-5t5nf-vnx6r"]
Oct 31 00:30:03.199875 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:03.199843    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" path="/var/lib/kubelet/pods/3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971/volumes"
Oct 31 00:30:05.246318 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:05.246263    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:05.411046 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:05.411007    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:09.049328 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.049297    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:30:09.049756 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.049592    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" containerName="insights-gathering"
Oct 31 00:30:09.049756 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.049605    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" containerName="insights-gathering"
Oct 31 00:30:09.049756 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.049650    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3e94f4-0e2f-4c2d-b9c1-f44a35ca4971" containerName="insights-gathering"
Oct 31 00:30:09.070476 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.070455    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153722 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153692    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153866 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153728    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153866 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153866 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153845    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153997 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153880    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153997 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153921    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.153997 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.153984    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.154129 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.154005    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dhs\" (UniqueName: \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.164315 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.164292    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:30:09.250665 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.250635    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:09.254847 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.254825    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.254961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.254859    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.254961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.254885    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255061 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255007    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255108 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255063    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255215 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255126    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255215 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255162    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dhs\" (UniqueName: \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255222    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255252    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255671 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255644    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255726    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.255835 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.255804    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.256073 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.256052    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.257865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.257843    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.257952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.257904    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.270566 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.270536    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dhs\" (UniqueName: \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\") pod \"console-5659588768-gw8md\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") " pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.385452 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.385392    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:09.399390 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.399368    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:09.542889 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.542856    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:09.684369 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.684315    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:09.702065 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:09.702046    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:30:09.704088 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:30:09.704058    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b198188_ab23_4179_bc2b_39065732f15b.slice/crio-ae77587e2d8e0196b2004721d82418dadf6c6804bfb02b7673bd167df6ecc245 WatchSource:0}: Error finding container ae77587e2d8e0196b2004721d82418dadf6c6804bfb02b7673bd167df6ecc245: Status 404 returned error can't find the container with id ae77587e2d8e0196b2004721d82418dadf6c6804bfb02b7673bd167df6ecc245
Oct 31 00:30:10.569069 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:10.569029    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5659588768-gw8md" event={"ID":"2b198188-ab23-4179-bc2b-39065732f15b","Type":"ContainerStarted","Data":"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"}
Oct 31 00:30:10.569585 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:10.569129    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5659588768-gw8md" event={"ID":"2b198188-ab23-4179-bc2b-39065732f15b","Type":"ContainerStarted","Data":"ae77587e2d8e0196b2004721d82418dadf6c6804bfb02b7673bd167df6ecc245"}
Oct 31 00:30:10.594282 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:10.594234    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5659588768-gw8md" podStartSLOduration=1.594221611 podStartE2EDuration="1.594221611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:10.594086951 +0000 UTC m=+73.503089604" watchObservedRunningTime="2026-10-31 00:30:10.594221611 +0000 UTC m=+73.503224264"
Oct 31 00:30:19.399982 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:19.399946    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:19.400589 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:19.400007    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:19.404774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:19.404751    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:19.603373 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:19.603340    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:30:19.667605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:19.667580    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:30:27.129576 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:30:27.129534    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:30:44.686977 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:44.686938    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58ffb96544-q5skl" podUID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" containerName="console" containerID="cri-o://14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd" gracePeriod=15
Oct 31 00:30:44.929687 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:44.929648    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:30:44.964017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:44.963958    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:45.006328 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006301    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006451 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006353    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006451 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006379    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006529 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006483    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006578 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006536    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006578 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006565    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006652 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006605    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\" (UniqueName: \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006722 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006664    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\" (UniqueName: \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\") pod \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\" (UID: \"169ffc8a-3df4-4e5f-9d46-6d03393ed396\") "
Oct 31 00:30:45.006765 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006727    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:45.006925 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006859    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:45.006925 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006897    4553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-oauth-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.007092 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.006986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:45.007138 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.007108    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:45.007226 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.007128    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:45.008892 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.008871    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "kube-api-access-zx2v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:45.008994 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.008906    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:45.009150 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.009126    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config" pod "169ffc8a-3df4-4e5f-9d46-6d03393ed396" (UID: "169ffc8a-3df4-4e5f-9d46-6d03393ed396"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:45.107330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107300    4553 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/169ffc8a-3df4-4e5f-9d46-6d03393ed396-tmp\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107324    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-service-ca\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107335    4553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107554 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107345    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-trusted-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107554 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107354    4553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-oauth-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107554 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107362    4553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169ffc8a-3df4-4e5f-9d46-6d03393ed396-console-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.107554 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.107370    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2v8\" (UniqueName: \"kubernetes.io/projected/169ffc8a-3df4-4e5f-9d46-6d03393ed396-kube-api-access-zx2v8\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.685403 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.685366    4553 generic.go:356] "Generic (PLEG): container finished" podID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" containerID="14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd" exitCode=0
Oct 31 00:30:45.685603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.685417    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ffb96544-q5skl"
Oct 31 00:30:45.685603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.685439    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ffb96544-q5skl" event={"ID":"169ffc8a-3df4-4e5f-9d46-6d03393ed396","Type":"ContainerDied","Data":"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"}
Oct 31 00:30:45.685603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.685479    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ffb96544-q5skl" event={"ID":"169ffc8a-3df4-4e5f-9d46-6d03393ed396","Type":"ContainerDied","Data":"786ffd7ab8037db7d2b3b54d47b6129c58aad497a431c19644153427062df163"}
Oct 31 00:30:45.685603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.685498    4553 scope.go:122] "RemoveContainer" containerID="14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"
Oct 31 00:30:45.698393 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.698373    4553 scope.go:122] "RemoveContainer" containerID="14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"
Oct 31 00:30:45.698774 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:30:45.698619    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd\": container with ID starting with 14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd not found: ID does not exist" containerID="14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"
Oct 31 00:30:45.698774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.698640    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd"} err="failed to get container status \"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd\": rpc error: code = NotFound desc = could not find container \"14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd\": container with ID starting with 14833038b9f1e1be4993ace5ace5225ac45a0c6b556afe3928c4a1e5159504bd not found: ID does not exist"
Oct 31 00:30:45.709292 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.709273    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:30:45.717556 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:45.717536    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58ffb96544-q5skl"]
Oct 31 00:30:46.119178 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:46.119104    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:47.198586 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:47.198551    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" path="/var/lib/kubelet/pods/169ffc8a-3df4-4e5f-9d46-6d03393ed396/volumes"
Oct 31 00:30:53.314526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:53.314490    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:57.129211 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:30:57.129177    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:30:58.784280 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:58.784241    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:30:58.929826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:30:58.929790    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:04.428641 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:04.428599    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:05.239736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:05.239699    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:05.376567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:05.376529    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:07.271314 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.271281    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d4d58bbdf-tvzxk"]
Oct 31 00:31:07.271704 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.271540    4553 cpu_manager.go:414] "containerMap: removing container" podUID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" containerName="console"
Oct 31 00:31:07.271704 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.271550    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" containerName="console"
Oct 31 00:31:07.271704 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.271612    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="169ffc8a-3df4-4e5f-9d46-6d03393ed396" containerName="console"
Oct 31 00:31:07.274375 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.274354    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.287825 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.287803    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4d58bbdf-tvzxk"]
Oct 31 00:31:07.362899 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.362873    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-console-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363014 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.362902    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247590ce-7250-44d4-9534-634352b1cd92-tmp\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363014 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.362942    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-trusted-ca-bundle\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363014 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.363012    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363119 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.363045    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rt8x\" (UniqueName: \"kubernetes.io/projected/247590ce-7250-44d4-9534-634352b1cd92-kube-api-access-8rt8x\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363119 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.363084    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-oauth-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363180 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.363128    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-service-ca\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.363180 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.363145    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-oauth-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.412000 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.411967    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:07.464127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464106    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-console-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464131    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247590ce-7250-44d4-9534-634352b1cd92-tmp\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464164    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-trusted-ca-bundle\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464180    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464203    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rt8x\" (UniqueName: \"kubernetes.io/projected/247590ce-7250-44d4-9534-634352b1cd92-kube-api-access-8rt8x\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464231    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-oauth-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464259    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-service-ca\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464289    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-oauth-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464557 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464533    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247590ce-7250-44d4-9534-634352b1cd92-tmp\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.464990 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.464966    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-console-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.465091 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.465041    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-oauth-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.465091 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.465071    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-service-ca\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.465384 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.465363    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247590ce-7250-44d4-9534-634352b1cd92-trusted-ca-bundle\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.466853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.466827    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-serving-cert\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.466948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.466933    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247590ce-7250-44d4-9534-634352b1cd92-console-oauth-config\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.479136 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.479116    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rt8x\" (UniqueName: \"kubernetes.io/projected/247590ce-7250-44d4-9534-634352b1cd92-kube-api-access-8rt8x\") pod \"console-6d4d58bbdf-tvzxk\" (UID: \"247590ce-7250-44d4-9534-634352b1cd92\") " pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.560377 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.560327    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:07.585451 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.585428    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:07.866405 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:07.866327    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4d58bbdf-tvzxk"]
Oct 31 00:31:07.869189 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:31:07.869160    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247590ce_7250_44d4_9534_634352b1cd92.slice/crio-f61e89e5648f9ab8ece6d65b5703579ad24a1e54d32001adcd7fc6c7e68952ca WatchSource:0}: Error finding container f61e89e5648f9ab8ece6d65b5703579ad24a1e54d32001adcd7fc6c7e68952ca: Status 404 returned error can't find the container with id f61e89e5648f9ab8ece6d65b5703579ad24a1e54d32001adcd7fc6c7e68952ca
Oct 31 00:31:08.760719 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:08.760620    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d58bbdf-tvzxk" event={"ID":"247590ce-7250-44d4-9534-634352b1cd92","Type":"ContainerStarted","Data":"99214c0d6b9df0b9a642ea9808ab2d6eea1ad223c4463a180dc85cb4e97ca187"}
Oct 31 00:31:08.760719 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:08.760668    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d58bbdf-tvzxk" event={"ID":"247590ce-7250-44d4-9534-634352b1cd92","Type":"ContainerStarted","Data":"f61e89e5648f9ab8ece6d65b5703579ad24a1e54d32001adcd7fc6c7e68952ca"}
Oct 31 00:31:08.786058 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:08.786013    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6d4d58bbdf-tvzxk" podStartSLOduration=1.786000555 podStartE2EDuration="1.786000555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:08.785227984 +0000 UTC m=+131.694230641" watchObservedRunningTime="2026-10-31 00:31:08.786000555 +0000 UTC m=+131.695003208"
Oct 31 00:31:09.146341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:09.146260    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:09.288709 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:09.287744    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:17.585739 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:17.585706    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:17.586114 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:17.585773    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:17.591832 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:17.591812    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:17.794721 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:17.794670    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d4d58bbdf-tvzxk"
Oct 31 00:31:17.860522 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:17.860453    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:31:24.815946 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:24.815917    4553 generic.go:356] "Generic (PLEG): container finished" podID="4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" containerID="aaf4ea65657a245d9a114eb6afe500f63d16b75223f88aa9419319a7b7f5abbc" exitCode=0
Oct 31 00:31:24.816371 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:24.815979    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2qbts-74vm8" event={"ID":"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92","Type":"ContainerDied","Data":"aaf4ea65657a245d9a114eb6afe500f63d16b75223f88aa9419319a7b7f5abbc"}
Oct 31 00:31:25.945464 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.945440    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:31:25.995937 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.995909    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\") pod \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") "
Oct 31 00:31:25.996075 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.995984    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\") pod \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") "
Oct 31 00:31:25.996075 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.996056    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\" (UniqueName: \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\") pod \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") "
Oct 31 00:31:25.996158 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.996086    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\") pod \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\" (UID: \"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92\") "
Oct 31 00:31:25.996299 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.996278    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path" pod "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" (UID: "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.996426 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.996401    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle" pod "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" (UID: "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:25.996553 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.996533    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle" pod "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" (UID: "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:25.998234 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:25.998214    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf" pod "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" (UID: "4cb2c039-eb2d-4b2e-bdc0-5c446e609a92"). InnerVolumeSpecName "kube-api-access-84vcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:26.097435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.097367    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-service-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:26.097435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.097390    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vcf\" (UniqueName: \"kubernetes.io/projected/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-kube-api-access-84vcf\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:26.097435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.097401    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-trusted-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:26.097435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.097410    4553 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb2c039-eb2d-4b2e-bdc0-5c446e609a92-archives-path\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:26.824457 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.824420    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2qbts-74vm8" event={"ID":"4cb2c039-eb2d-4b2e-bdc0-5c446e609a92","Type":"ContainerDied","Data":"97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb"}
Oct 31 00:31:26.824457 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.824453    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2qbts-74vm8"
Oct 31 00:31:26.824659 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:26.824472    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97600717af60f20e8efd707c3f47b27d4eaadc3395e0cf9760487d297970edeb"
Oct 31 00:31:27.128496 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:31:27.128417    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:31:27.242194 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:27.242162    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:27.380116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:27.380034    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:32.085147 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:32.085114    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:42.885703 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:42.885625    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5659588768-gw8md" podUID="2b198188-ab23-4179-bc2b-39065732f15b" containerName="console" containerID="cri-o://6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f" gracePeriod=15
Oct 31 00:31:43.121321 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.121296    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:31:43.217327 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217302    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217467 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217338    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217467 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217366    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217467 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217418    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217467 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217446    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217477    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217516    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217565    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\" (UniqueName: \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\") pod \"2b198188-ab23-4179-bc2b-39065732f15b\" (UID: \"2b198188-ab23-4179-bc2b-39065732f15b\") "
Oct 31 00:31:43.217657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217586    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:43.217899 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217879    4553 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b198188-ab23-4179-bc2b-39065732f15b-tmp\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.217960 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217873    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:43.217960 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217883    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:43.217960 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217921    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:43.218117 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.217975    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:43.219707 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.219659    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:43.220023 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.220002    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "kube-api-access-89dhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:43.220083 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.220031    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config" pod "2b198188-ab23-4179-bc2b-39065732f15b" (UID: "2b198188-ab23-4179-bc2b-39065732f15b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:43.319133 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319104    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dhs\" (UniqueName: \"kubernetes.io/projected/2b198188-ab23-4179-bc2b-39065732f15b-kube-api-access-89dhs\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319133 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319128    4553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319142    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-trusted-ca-bundle\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319154    4553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-oauth-serving-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319167    4553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b198188-ab23-4179-bc2b-39065732f15b-console-oauth-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319180    4553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-console-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.319287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.319192    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b198188-ab23-4179-bc2b-39065732f15b-service-ca\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:43.880995 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.880959    4553 generic.go:356] "Generic (PLEG): container finished" podID="2b198188-ab23-4179-bc2b-39065732f15b" containerID="6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f" exitCode=0
Oct 31 00:31:43.881170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.881007    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5659588768-gw8md"
Oct 31 00:31:43.881170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.881047    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5659588768-gw8md" event={"ID":"2b198188-ab23-4179-bc2b-39065732f15b","Type":"ContainerDied","Data":"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"}
Oct 31 00:31:43.881170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.881103    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5659588768-gw8md" event={"ID":"2b198188-ab23-4179-bc2b-39065732f15b","Type":"ContainerDied","Data":"ae77587e2d8e0196b2004721d82418dadf6c6804bfb02b7673bd167df6ecc245"}
Oct 31 00:31:43.881170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.881124    4553 scope.go:122] "RemoveContainer" containerID="6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"
Oct 31 00:31:43.892726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.892525    4553 scope.go:122] "RemoveContainer" containerID="6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"
Oct 31 00:31:43.892957 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:31:43.892753    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f\": container with ID starting with 6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f not found: ID does not exist" containerID="6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"
Oct 31 00:31:43.892957 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.892777    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f"} err="failed to get container status \"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f\": rpc error: code = NotFound desc = could not find container \"6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f\": container with ID starting with 6b57c01a31a348c11e272a807984d5728b98e66806d1202a4f32840ca0d06a5f not found: ID does not exist"
Oct 31 00:31:43.907719 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.907698    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:31:43.914218 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:43.914202    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5659588768-gw8md"]
Oct 31 00:31:45.040940 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:45.040888    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:31:45.198885 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:31:45.198854    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b198188-ab23-4179-bc2b-39065732f15b" path="/var/lib/kubelet/pods/2b198188-ab23-4179-bc2b-39065732f15b/volumes"
Oct 31 00:31:57.128966 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:31:57.128925    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:32:00.800081 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:32:00.800045    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:32:00.934446 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:32:00.934410    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:32:27.128697 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:32:27.128648    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:32:45.095702 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:32:45.095593    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:32:57.128980 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:32:57.128940    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:33:27.129410 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:33:27.129372    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:33:37.675755 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:33:37.675712    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:33:38.960699 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:33:38.960649    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:33:45.146192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:33:45.146147    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:33:57.129323 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:33:57.129290    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:33:57.207397 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:33:57.207367    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:33:57.208797 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:33:57.208776    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:34:12.007749 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:12.007643    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:12.145237 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:12.145205    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:12.279933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:12.279859    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:15.658271 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:15.658238    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:15.796354 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:15.796304    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:27.129235 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:34:27.129197    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:34:40.591456 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:40.591423    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:40.744478 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:40.744443    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:45.211388 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:45.211346    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:46.664309 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:46.664266    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:47.013882 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.013806    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:47.159650 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.159620    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:47.290874 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.290789    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:47.449961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.449925    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"]
Oct 31 00:34:47.450265 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450247    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" containerName="insights-gathering"
Oct 31 00:34:47.450351 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450267    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" containerName="insights-gathering"
Oct 31 00:34:47.450351 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450280    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2b198188-ab23-4179-bc2b-39065732f15b" containerName="console"
Oct 31 00:34:47.450351 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450288    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b198188-ab23-4179-bc2b-39065732f15b" containerName="console"
Oct 31 00:34:47.450494 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450353    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb2c039-eb2d-4b2e-bdc0-5c446e609a92" containerName="insights-gathering"
Oct 31 00:34:47.450494 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.450363    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b198188-ab23-4179-bc2b-39065732f15b" containerName="console"
Oct 31 00:34:47.453250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.453231    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.459853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.459728    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 31 00:34:47.459940 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.459918    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 31 00:34:47.460015 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.459986    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 31 00:34:47.460369 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.460345    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8tff9\""
Oct 31 00:34:47.460477 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.460416    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 31 00:34:47.460477 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.460426    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 31 00:34:47.465091 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.465068    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"]
Oct 31 00:34:47.557945 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.557861    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.557945 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.557899    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2dh\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-kube-api-access-xb2dh\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.557945 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.557918    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a466c238-0d4c-47b8-9449-d50b6bc0aae9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.659218 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.659190    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.659218 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.659226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2dh\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-kube-api-access-xb2dh\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.659449 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.659256    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a466c238-0d4c-47b8-9449-d50b6bc0aae9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.659729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.659706    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a466c238-0d4c-47b8-9449-d50b6bc0aae9-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.661888 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.661870    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.675241 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.675220    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2dh\" (UniqueName: \"kubernetes.io/projected/a466c238-0d4c-47b8-9449-d50b6bc0aae9-kube-api-access-xb2dh\") pod \"keda-metrics-apiserver-86bd856cfb-m7whv\" (UID: \"a466c238-0d4c-47b8-9449-d50b6bc0aae9\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.766398 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.766300    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:47.767525 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.767504    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-lvzxd"]
Oct 31 00:34:47.770473 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.770452    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.781812 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.781786    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 31 00:34:47.782765 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.782742    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-lvzxd"]
Oct 31 00:34:47.823275 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.823236    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:47.861838 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.861621    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-certificates\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.861838 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.861669    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq6v\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-kube-api-access-swq6v\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.963137 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.963101    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-certificates\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.963137 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.963137    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq6v\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-kube-api-access-swq6v\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.965890 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.965866    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-certificates\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:47.987452 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:47.987414    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq6v\" (UniqueName: \"kubernetes.io/projected/f19a1589-65de-427c-a5e7-0117270e7a52-kube-api-access-swq6v\") pod \"keda-admission-7b7c985c58-lvzxd\" (UID: \"f19a1589-65de-427c-a5e7-0117270e7a52\") " pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:48.015740 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.015709    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:48.062879 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.062721    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"]
Oct 31 00:34:48.065608 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:34:48.065583    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda466c238_0d4c_47b8_9449_d50b6bc0aae9.slice/crio-b3562f024f702a435cc89a2a3ec8a4d8945d4a1cb16939e7e51290ffdf1108ab WatchSource:0}: Error finding container b3562f024f702a435cc89a2a3ec8a4d8945d4a1cb16939e7e51290ffdf1108ab: Status 404 returned error can't find the container with id b3562f024f702a435cc89a2a3ec8a4d8945d4a1cb16939e7e51290ffdf1108ab
Oct 31 00:34:48.067007 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.066987    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:34:48.101544 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.101471    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:48.340326 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.340279    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:48.389499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.389418    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-lvzxd"]
Oct 31 00:34:48.394893 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:34:48.394868    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19a1589_65de_427c_a5e7_0117270e7a52.slice/crio-9c31227620a5017d9a708fdc94ef6487aaedb92c6d343e270e98916747595aef WatchSource:0}: Error finding container 9c31227620a5017d9a708fdc94ef6487aaedb92c6d343e270e98916747595aef: Status 404 returned error can't find the container with id 9c31227620a5017d9a708fdc94ef6487aaedb92c6d343e270e98916747595aef
Oct 31 00:34:48.465756 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.465717    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd" event={"ID":"f19a1589-65de-427c-a5e7-0117270e7a52","Type":"ContainerStarted","Data":"9c31227620a5017d9a708fdc94ef6487aaedb92c6d343e270e98916747595aef"}
Oct 31 00:34:48.466954 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.466926    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv" event={"ID":"a466c238-0d4c-47b8-9449-d50b6bc0aae9","Type":"ContainerStarted","Data":"b3562f024f702a435cc89a2a3ec8a4d8945d4a1cb16939e7e51290ffdf1108ab"}
Oct 31 00:34:48.645044 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.644959    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:48.781345 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:48.781308    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:34:52.486211 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.486173    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv" event={"ID":"a466c238-0d4c-47b8-9449-d50b6bc0aae9","Type":"ContainerStarted","Data":"0755434b0e92efda77e31b16f67281040e4957e8ba0a94685be897e3c8ccb776"}
Oct 31 00:34:52.486617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.486342    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:34:52.487859 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.487822    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd" event={"ID":"f19a1589-65de-427c-a5e7-0117270e7a52","Type":"ContainerStarted","Data":"53da9b446177b52c5adaa3f3acc980dfa43b7962404ff9b4b410eb15d712f5b5"}
Oct 31 00:34:52.488011 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.487953    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:34:52.511127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.511079    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv" podStartSLOduration=1.8488814759999999 podStartE2EDuration="5.511067132s" totalImagesPullingTime="3.662185656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:52.510425484 +0000 UTC m=+355.419428150" watchObservedRunningTime="2026-10-31 00:34:52.511067132 +0000 UTC m=+355.420069834"
Oct 31 00:34:52.533814 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:34:52.533774    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd" podStartSLOduration=2.3135855850000002 podStartE2EDuration="5.533762372s" totalImagesPullingTime="3.220176787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:52.533464213 +0000 UTC m=+355.442466884" watchObservedRunningTime="2026-10-31 00:34:52.533762372 +0000 UTC m=+355.442765048"
Oct 31 00:34:57.132347 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:34:57.132308    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:35:03.495841 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:03.495813    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-m7whv"
Oct 31 00:35:13.493347 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:13.493319    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-lvzxd"
Oct 31 00:35:27.128961 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:35:27.128923    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:35:40.938169 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:40.938084    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:35:41.093540 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:41.093496    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:35:45.265723 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:45.265665    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:35:45.434838 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:35:45.434794    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:35:57.130063 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:35:57.130025    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:36:17.922317 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:17.922279    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:18.073586 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.073552    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:18.131451 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.131423    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-ggkf9"]
Oct 31 00:36:18.134574 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.134556    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.145341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.145314    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-ggkf9"]
Oct 31 00:36:18.164398 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.164374    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 00:36:18.164664 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.164649    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 00:36:18.166207 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.166194    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 31 00:36:18.166259 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.166206    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f8d7d\""
Oct 31 00:36:18.211808 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.211741    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:18.217776 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.217758    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6vd\" (UniqueName: \"kubernetes.io/projected/31742dd9-374b-4a4f-846b-7e148923c08f-kube-api-access-np6vd\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.217830 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.217789    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31742dd9-374b-4a4f-846b-7e148923c08f-data\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.318448 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.318421    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6vd\" (UniqueName: \"kubernetes.io/projected/31742dd9-374b-4a4f-846b-7e148923c08f-kube-api-access-np6vd\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.318566 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.318454    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31742dd9-374b-4a4f-846b-7e148923c08f-data\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.318855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.318840    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31742dd9-374b-4a4f-846b-7e148923c08f-data\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.334513 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.334486    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6vd\" (UniqueName: \"kubernetes.io/projected/31742dd9-374b-4a4f-846b-7e148923c08f-kube-api-access-np6vd\") pod \"seaweedfs-78bb9c6ff8-ggkf9\" (UID: \"31742dd9-374b-4a4f-846b-7e148923c08f\") " pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.353033 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.353000    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:18.445380 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.445358    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:18.718419 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.718394    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-ggkf9"]
Oct 31 00:36:18.720367 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:36:18.720341    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31742dd9_374b_4a4f_846b_7e148923c08f.slice/crio-fa33ca6f789bd023d1f1c8498382558427758264b3eaeefa372b5afce9048ee2 WatchSource:0}: Error finding container fa33ca6f789bd023d1f1c8498382558427758264b3eaeefa372b5afce9048ee2: Status 404 returned error can't find the container with id fa33ca6f789bd023d1f1c8498382558427758264b3eaeefa372b5afce9048ee2
Oct 31 00:36:18.785168 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:18.785093    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9" event={"ID":"31742dd9-374b-4a4f-846b-7e148923c08f","Type":"ContainerStarted","Data":"fa33ca6f789bd023d1f1c8498382558427758264b3eaeefa372b5afce9048ee2"}
Oct 31 00:36:19.648292 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:19.648250    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:19.782611 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:19.782570    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:19.917528 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:19.917500    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:27.132825 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:36:27.132794    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:36:28.823357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:28.823312    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9" event={"ID":"31742dd9-374b-4a4f-846b-7e148923c08f","Type":"ContainerStarted","Data":"570494c58a0976721b88d4c3da5787ca3b4cc32f180ed175c7a7717651ab03cc"}
Oct 31 00:36:28.823783 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:28.823390    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:28.846029 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:28.845981    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9" podStartSLOduration=1.636348118 podStartE2EDuration="10.845967434s" totalImagesPullingTime="9.209619316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:28.845273845 +0000 UTC m=+451.754276500" watchObservedRunningTime="2026-10-31 00:36:28.845967434 +0000 UTC m=+451.754970137"
Oct 31 00:36:34.831808 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:34.831776    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-ggkf9"
Oct 31 00:36:45.328049 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:36:45.328010    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:36:57.129185 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:36:57.129137    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:37:27.130060 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:37:27.129974    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:37:40.720393 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.720348    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:40.831110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.831077    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-wm8d8"]
Oct 31 00:37:40.834149 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.834133    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:40.843846 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.843825    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 31 00:37:40.843846 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.843826    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-nkfxz\""
Oct 31 00:37:40.850835 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.850812    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-wm8d8"]
Oct 31 00:37:40.874355 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.874329    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:40.881982 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.881962    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-z4k4j"]
Oct 31 00:37:40.884952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.884937    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:40.900543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.900511    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-z4k4j"]
Oct 31 00:37:40.904928 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.904906    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea0029f-543e-4002-a06e-30330910b046-tls-certs\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:40.905016 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.904936    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbl6\" (UniqueName: \"kubernetes.io/projected/1ea0029f-543e-4002-a06e-30330910b046-kube-api-access-gzbl6\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:40.942625 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.942597    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6czr4\""
Oct 31 00:37:40.942743 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.942715    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 31 00:37:40.942804 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:40.942787    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 31 00:37:41.006070 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.005995    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.006070 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.006038    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea0029f-543e-4002-a06e-30330910b046-tls-certs\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:41.006070 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.006066    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncht\" (UniqueName: \"kubernetes.io/projected/c12012ed-939e-4297-a548-8d688891babd-kube-api-access-cncht\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.006254 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.006152    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbl6\" (UniqueName: \"kubernetes.io/projected/1ea0029f-543e-4002-a06e-30330910b046-kube-api-access-gzbl6\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:41.006254 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.006193    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-cert\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.008632 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.008610    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea0029f-543e-4002-a06e-30330910b046-tls-certs\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:41.011033 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.010992    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:41.021301 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.021280    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbl6\" (UniqueName: \"kubernetes.io/projected/1ea0029f-543e-4002-a06e-30330910b046-kube-api-access-gzbl6\") pod \"model-serving-api-56ddc4c448-wm8d8\" (UID: \"1ea0029f-543e-4002-a06e-30330910b046\") " pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:41.106722 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.106694    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.106849 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.106731    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncht\" (UniqueName: \"kubernetes.io/projected/c12012ed-939e-4297-a548-8d688891babd-kube-api-access-cncht\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.106912 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.106862    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-cert\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.109108 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.109089    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.109196 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.109130    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12012ed-939e-4297-a548-8d688891babd-cert\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.121610 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.121590    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncht\" (UniqueName: \"kubernetes.io/projected/c12012ed-939e-4297-a548-8d688891babd-kube-api-access-cncht\") pod \"odh-model-controller-7f9f8cdc65-z4k4j\" (UID: \"c12012ed-939e-4297-a548-8d688891babd\") " pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.146627 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.146605    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:41.202855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.202826    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:41.421164 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.421136    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-wm8d8"]
Oct 31 00:37:41.423624 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:37:41.423593    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea0029f_543e_4002_a06e_30330910b046.slice/crio-86fb95a818a81fdaf8c0fe325b30e95d973385d772429ef6220109db8299b8d7 WatchSource:0}: Error finding container 86fb95a818a81fdaf8c0fe325b30e95d973385d772429ef6220109db8299b8d7: Status 404 returned error can't find the container with id 86fb95a818a81fdaf8c0fe325b30e95d973385d772429ef6220109db8299b8d7
Oct 31 00:37:41.481413 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:41.481388    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-z4k4j"]
Oct 31 00:37:41.483237 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:37:41.483207    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12012ed_939e_4297_a548_8d688891babd.slice/crio-a8d3cd0aa1a53da9a330bfe4288567a41d6ea39759cda6b9121d40f088737838 WatchSource:0}: Error finding container a8d3cd0aa1a53da9a330bfe4288567a41d6ea39759cda6b9121d40f088737838: Status 404 returned error can't find the container with id a8d3cd0aa1a53da9a330bfe4288567a41d6ea39759cda6b9121d40f088737838
Oct 31 00:37:42.068308 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:42.068268    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-wm8d8" event={"ID":"1ea0029f-543e-4002-a06e-30330910b046","Type":"ContainerStarted","Data":"86fb95a818a81fdaf8c0fe325b30e95d973385d772429ef6220109db8299b8d7"}
Oct 31 00:37:42.069809 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:42.069742    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j" event={"ID":"c12012ed-939e-4297-a548-8d688891babd","Type":"ContainerStarted","Data":"a8d3cd0aa1a53da9a330bfe4288567a41d6ea39759cda6b9121d40f088737838"}
Oct 31 00:37:44.530621 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:44.530583    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:44.672401 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:44.672363    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:45.083109 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.083024    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-wm8d8" event={"ID":"1ea0029f-543e-4002-a06e-30330910b046","Type":"ContainerStarted","Data":"2a43cf398d72fa12cca7d9928495fc64c7b1a2ad756d7888ff98b2c6c63d44ce"}
Oct 31 00:37:45.083284 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.083233    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:45.084736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.084708    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j" event={"ID":"c12012ed-939e-4297-a548-8d688891babd","Type":"ContainerStarted","Data":"fcf51951c5df65a061db713afa5e7b8770087c6c87626661867520d549448530"}
Oct 31 00:37:45.084871 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.084763    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:45.106253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.106207    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-wm8d8" podStartSLOduration=2.4163623149999998 podStartE2EDuration="5.106195949s" totalImagesPullingTime="2.689833634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:45.105739402 +0000 UTC m=+528.014742058" watchObservedRunningTime="2026-10-31 00:37:45.106195949 +0000 UTC m=+528.015198602"
Oct 31 00:37:45.128813 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.128768    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j" podStartSLOduration=2.272328868 podStartE2EDuration="5.128756955s" totalImagesPullingTime="2.856428087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:45.128619744 +0000 UTC m=+528.037622411" watchObservedRunningTime="2026-10-31 00:37:45.128756955 +0000 UTC m=+528.037759609"
Oct 31 00:37:45.389737 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:45.389648    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:37:56.090783 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:56.090756    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-z4k4j"
Oct 31 00:37:56.092665 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:37:56.092648    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-wm8d8"
Oct 31 00:37:57.129924 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:37:57.129882    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:38:27.128522 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:38:27.128485    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:38:45.452312 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:38:45.452226    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:38:57.129642 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:38:57.129606    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:38:57.227717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:38:57.227670    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:38:57.230267 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:38:57.230246    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:39:02.444051 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:39:02.443996    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:39:17.165443 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:39:17.165409    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:39:27.129533 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:39:27.129491    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:39:29.529470 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:39:29.529432    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:39:29.681032 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:39:29.680993    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:39:45.505893 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:39:45.505851    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:39:57.130383 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:39:57.130347    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:40:05.191763 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:05.191720    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:05.332994 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:05.332960    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:05.470825 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:05.470749    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:18.841781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:18.841664    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:18.977831 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:18.977801    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:27.129139 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:40:27.129100    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:40:38.466984 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:38.466941    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:38.614609 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:38.614571    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:39.758916 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:39.758869    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:39.905372 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:39.905338    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:40.052554 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:40.052470    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:40.200548 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:40.200519    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:45.598470 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:40:45.598434    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:40:57.128900 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:40:57.128855    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:41:27.129755 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:41:27.129710    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:41:28.048920 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:41:28.048882    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:41:45.638778 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:41:45.638704    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:41:57.129576 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:41:57.129540    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:42:27.129442 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:42:27.129404    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:42:35.738049 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:42:35.738009    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:42:41.682452 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:42:41.682417    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:42:45.686662 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:42:45.686619    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:42:47.070736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:42:47.070703    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:42:47.212340 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:42:47.212304    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:42:57.129122 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:42:57.129089    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:43:07.434392 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:43:07.434358    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:43:07.576428 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:43:07.576398    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:43:27.129017 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:43:27.128935    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:43:45.754154 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:43:45.754095    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:43:57.129432 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:43:57.129396    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:43:57.251603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:43:57.251577    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:43:57.254671 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:43:57.254650    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:44:27.130145 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:44:27.130106    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:44:32.587924 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:32.587885    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:34.195665 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:34.195634    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:43.282272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:43.282188    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:45.797208 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:45.797172    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:50.752518 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:50.752480    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:50.888300 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:50.888266    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:56.418202 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:56.418155    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:56.557841 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:44:56.557789    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:44:57.129803 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:44:57.129771    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:45:01.616811 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:01.616772    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:06.544993 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:06.544955    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:09.165345 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:09.165306    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:09.305066 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:09.305030    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:15.013348 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.013306    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:15.104284 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.104251    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:45:15.107848 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.107828    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:15.117012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.116991    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:45:15.117290 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.117269    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:45:15.117290 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.117276    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:45:15.117462 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.117295    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:45:15.155156 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.155120    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdrd\" (UniqueName: \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\") pod \"proxy-routing-d2160853-sink-5bf48bbf54-94zfs\" (UID: \"d0319cf0-3b3f-4f06-abda-2441481e376d\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:15.193193 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.193157    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:15.256095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.256062    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdrd\" (UniqueName: \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\") pod \"proxy-routing-d2160853-sink-5bf48bbf54-94zfs\" (UID: \"d0319cf0-3b3f-4f06-abda-2441481e376d\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:15.271517 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.271449    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdrd\" (UniqueName: \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\") pod \"proxy-routing-d2160853-sink-5bf48bbf54-94zfs\" (UID: \"d0319cf0-3b3f-4f06-abda-2441481e376d\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:15.421865 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.421823    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:15.700928 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.700898    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:45:15.703889 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:45:15.703861    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0319cf0_3b3f_4f06_abda_2441481e376d.slice/crio-6728c8e61ac40a5d974c235b433b942b83b83053018a487b5b9b574812a44e2e WatchSource:0}: Error finding container 6728c8e61ac40a5d974c235b433b942b83b83053018a487b5b9b574812a44e2e: Status 404 returned error can't find the container with id 6728c8e61ac40a5d974c235b433b942b83b83053018a487b5b9b574812a44e2e
Oct 31 00:45:15.706335 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:15.706315    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:45:16.643880 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:16.643844    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" event={"ID":"d0319cf0-3b3f-4f06-abda-2441481e376d","Type":"ContainerStarted","Data":"6728c8e61ac40a5d974c235b433b942b83b83053018a487b5b9b574812a44e2e"}
Oct 31 00:45:18.652512 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:18.652471    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" event={"ID":"d0319cf0-3b3f-4f06-abda-2441481e376d","Type":"ContainerStarted","Data":"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"}
Oct 31 00:45:18.652927 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:18.652601    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:18.654933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:18.654915    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:45:18.674376 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:18.674331    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" podStartSLOduration=1.627244991 podStartE2EDuration="3.674319942s" totalImagesPullingTime="2.047074951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:18.673368319 +0000 UTC m=+981.582370974" watchObservedRunningTime="2026-10-31 00:45:18.674319942 +0000 UTC m=+981.583322596"
Oct 31 00:45:19.529384 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:19.529349    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:19.671923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:19.671886    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:22.057792 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.057755    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:22.199183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.199146    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:22.512669 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.512639    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:45:22.516067 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.516051    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.522073 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.522046    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-d2160853-kube-rbac-proxy-sar-config\""
Oct 31 00:45:22.522816 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.522797    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-d2160853-dockercfg-gwv8s\""
Oct 31 00:45:22.522893 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.522822    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-d2160853-predictor-serving-cert\""
Oct 31 00:45:22.529336 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.529316    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:45:22.613166 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.613146    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.613277 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.613179    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.613277 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.613197    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmjs\" (UniqueName: \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.678289 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.678265    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:22.714099 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.714073    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.714202 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.714106    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmjs\" (UniqueName: \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.714202 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.714175    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.714800 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.714784    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.716594 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.716575    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:22.729247 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:22.729226    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmjs\" (UniqueName: \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:27.129476 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:45:27.129441    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:45:27.748972 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:27.748932    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:27.758281 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:27.758254    4553 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:27.850011 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:27.849980    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:27.850181 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:27.850116    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:27.854842 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:27.854825    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") DevicePath \"csi-5dbef36f7953750475b64f5362d5846ba76274c305650c6a8bde0b40d1cc7b60\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:28.030304 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.030265    4553 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bf2d1d6ae9837ffbdfe16af9d8111147737743a7c3a4f2cef682f9bea8c6a5a/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:28.045139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.045106    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-6ffc787844-9hdfh\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:28.229298 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.229249    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:28.514062 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.514038    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:45:28.516222 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:45:28.516193    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1 WatchSource:0}: Error finding container dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1: Status 404 returned error can't find the container with id dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1
Oct 31 00:45:28.690585 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.690551    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerStarted","Data":"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"}
Oct 31 00:45:28.690747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:28.690613    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerStarted","Data":"dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1"}
Oct 31 00:45:31.704777 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:31.704739    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerStarted","Data":"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"}
Oct 31 00:45:31.705251 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:31.704924    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:31.731100 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:31.731060    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podStartSLOduration=10.051184462 podStartE2EDuration="12.731044393s" totalImagesPullingTime="2.679859931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:31.730117405 +0000 UTC m=+994.639120059" watchObservedRunningTime="2026-10-31 00:45:31.731044393 +0000 UTC m=+994.640047048"
Oct 31 00:45:32.708838 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:32.708806    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:32.710966 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:32.710945    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:39.720858 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:39.720828    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:45:45.861295 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:45.861260    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:45:49.633729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.633694    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:45:49.637177 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.637159    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.649899 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.649876    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:45:49.713066 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.713036    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlj5k\" (UniqueName: \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.713225 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.713099    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.713225 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.713174    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.813965 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.813930    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.814146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.813974    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlj5k\" (UniqueName: \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.814146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.814111    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.814259 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.814162    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.814313 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.814278    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") DevicePath \"csi-5dbef36f7953750475b64f5362d5846ba76274c305650c6a8bde0b40d1cc7b60\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.814769 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.814750    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.816874 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.816856    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.818772 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.818756    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") DevicePath \"csi-5dbef36f7953750475b64f5362d5846ba76274c305650c6a8bde0b40d1cc7b60\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.829173 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.829148    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlj5k\" (UniqueName: \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.837536 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.837501    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"proxy-routing-d2160853-predictor-94566c6b9-wvlpw\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:49.952432 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:49.952406    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:50.228797 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:50.228780    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:45:50.231013 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:45:50.230982    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ba5b3_ad15_4c46_86a1_c0cf049f98a3.slice/crio-194cf466a9b19b65571be84f7af889c81b1b11f0bc7eb501496aac3df5f849c4 WatchSource:0}: Error finding container 194cf466a9b19b65571be84f7af889c81b1b11f0bc7eb501496aac3df5f849c4: Status 404 returned error can't find the container with id 194cf466a9b19b65571be84f7af889c81b1b11f0bc7eb501496aac3df5f849c4
Oct 31 00:45:50.775128 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:50.775097    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerStarted","Data":"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"}
Oct 31 00:45:50.775510 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:50.775157    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerStarted","Data":"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"}
Oct 31 00:45:50.775510 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:50.775181    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerStarted","Data":"194cf466a9b19b65571be84f7af889c81b1b11f0bc7eb501496aac3df5f849c4"}
Oct 31 00:45:55.796915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.796877    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerStarted","Data":"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"}
Oct 31 00:45:55.797368 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.797090    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.797368 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.797132    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.797368 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.797153    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.799863 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.799840    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.800369 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.800350    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.802521 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.802505    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:45:55.826309 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.826267    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podStartSLOduration=2.415048427 podStartE2EDuration="6.826256096s" totalImagesPullingTime="4.411207669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:55.824451349 +0000 UTC m=+1018.733454004" watchObservedRunningTime="2026-10-31 00:45:55.826256096 +0000 UTC m=+1018.735258751"
Oct 31 00:45:55.890161 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.890134    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:45:55.890361 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.890341    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kserve-container" containerID="cri-o://00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771" gracePeriod=30
Oct 31 00:45:55.890477 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:55.890441    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" containerID="cri-o://0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d" gracePeriod=30
Oct 31 00:45:56.802048 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:56.802016    4553 generic.go:356] "Generic (PLEG): container finished" podID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerID="0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d" exitCode=0
Oct 31 00:45:56.802466 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:56.802083    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerDied","Data":"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"}
Oct 31 00:45:57.129637 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:45:57.129552    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:45:59.716784 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:45:59.716727    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:04.717220 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:04.717180    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:05.483282 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:05.483246    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:09.716883 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:09.716802    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:09.717282 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:09.716914    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:46:14.717425 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:14.717383    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:19.717105 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:19.717065    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:24.717038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:24.716996    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Oct 31 00:46:25.935014 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:25.934849    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-conmon-00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:46:25.940716 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:25.940664    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-conmon-00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:46:25.941363 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:25.941326    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9aaed_e22d_49f0_b98f_190e6980c441.slice/crio-conmon-00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:46:26.065196 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.065173    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:46:26.094116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.094088    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\") pod \"88e9aaed-e22d-49f0-b98f-190e6980c441\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") "
Oct 31 00:46:26.094251 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.094148    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\" (UniqueName: \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\") pod \"88e9aaed-e22d-49f0-b98f-190e6980c441\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") "
Oct 31 00:46:26.094323 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.094254    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"88e9aaed-e22d-49f0-b98f-190e6980c441\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") "
Oct 31 00:46:26.094323 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.094299    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\") pod \"88e9aaed-e22d-49f0-b98f-190e6980c441\" (UID: \"88e9aaed-e22d-49f0-b98f-190e6980c441\") "
Oct 31 00:46:26.094885 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.094859    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config" pod "88e9aaed-e22d-49f0-b98f-190e6980c441" (UID: "88e9aaed-e22d-49f0-b98f-190e6980c441"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:26.098276 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.098244    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls" pod "88e9aaed-e22d-49f0-b98f-190e6980c441" (UID: "88e9aaed-e22d-49f0-b98f-190e6980c441"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:26.098276 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.098265    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs" pod "88e9aaed-e22d-49f0-b98f-190e6980c441" (UID: "88e9aaed-e22d-49f0-b98f-190e6980c441"). InnerVolumeSpecName "kube-api-access-2vmjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:26.099003 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.098982    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78" pod "88e9aaed-e22d-49f0-b98f-190e6980c441" (UID: "88e9aaed-e22d-49f0-b98f-190e6980c441"). InnerVolumeSpecName "pvc-93933fae-9f0d-46de-b846-59ea91b29a09". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:46:26.194864 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.194838    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e9aaed-e22d-49f0-b98f-190e6980c441-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:26.194864 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.194865    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmjs\" (UniqueName: \"kubernetes.io/projected/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-api-access-2vmjs\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:26.195029 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.194876    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e9aaed-e22d-49f0-b98f-190e6980c441-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:26.917093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.917061    4553 generic.go:356] "Generic (PLEG): container finished" podID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerID="00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771" exitCode=137
Oct 31 00:46:26.917256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.917114    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"
Oct 31 00:46:26.917256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.917152    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerDied","Data":"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"}
Oct 31 00:46:26.917256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.917208    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh" event={"ID":"88e9aaed-e22d-49f0-b98f-190e6980c441","Type":"ContainerDied","Data":"dbe6b2e1d8febf6591ba13ede44c9ee45e9f4868f98c17bf905cccc3c6f722c1"}
Oct 31 00:46:26.917256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.917232    4553 scope.go:122] "RemoveContainer" containerID="0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"
Oct 31 00:46:26.925904 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.925890    4553 scope.go:122] "RemoveContainer" containerID="00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"
Oct 31 00:46:26.935193 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.935172    4553 scope.go:122] "RemoveContainer" containerID="0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"
Oct 31 00:46:26.935646 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:26.935622    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d\": container with ID starting with 0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d not found: ID does not exist" containerID="0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"
Oct 31 00:46:26.935741 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.935651    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d"} err="failed to get container status \"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d\": rpc error: code = NotFound desc = could not find container \"0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d\": container with ID starting with 0ed42df193d2eab25d7e62e88979c4126c7312301e339429c00350c7893fa45d not found: ID does not exist"
Oct 31 00:46:26.935741 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.935669    4553 scope.go:122] "RemoveContainer" containerID="00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"
Oct 31 00:46:26.935916 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:26.935896    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771\": container with ID starting with 00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771 not found: ID does not exist" containerID="00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"
Oct 31 00:46:26.935969 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.935922    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771"} err="failed to get container status \"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771\": rpc error: code = NotFound desc = could not find container \"00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771\": container with ID starting with 00ceefc0f2b63585f9cb95edc68243e02973ccc41c6e56bd245fd9c66243c771 not found: ID does not exist"
Oct 31 00:46:26.942432 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.942410    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:46:26.948839 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:26.948818    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-6ffc787844-9hdfh"]
Oct 31 00:46:27.130237 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:27.130201    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:46:27.199823 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:27.199750    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" path="/var/lib/kubelet/pods/88e9aaed-e22d-49f0-b98f-190e6980c441/volumes"
Oct 31 00:46:28.041746 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.041701    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-f4cda2dafd_proxy-routing-d2160853-sink-5bf48bbf54-94zfs_d0319cf0-3b3f-4f06-abda-2441481e376d/kserve-container/0.log"
Oct 31 00:46:28.141413 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.141380    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:46:28.141644 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.141619    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" podUID="d0319cf0-3b3f-4f06-abda-2441481e376d" containerName="kserve-container" containerID="cri-o://324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673" gracePeriod=30
Oct 31 00:46:28.194832 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.194804    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:28.257512 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.257476    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:46:28.257793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.257768    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kserve-container" containerID="cri-o://8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e" gracePeriod=30
Oct 31 00:46:28.257886 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.257859    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent" containerID="cri-o://4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b" gracePeriod=30
Oct 31 00:46:28.257969 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.257858    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" containerID="cri-o://d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd" gracePeriod=30
Oct 31 00:46:28.326139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.326066    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:28.929095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.929061    4553 generic.go:356] "Generic (PLEG): container finished" podID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerID="d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd" exitCode=0
Oct 31 00:46:28.929272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:28.929118    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerDied","Data":"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"}
Oct 31 00:46:30.797942 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:30.797896    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:35.797837 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:35.797798    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:35.798334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:35.798131    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:46:40.797765 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:40.797726    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:40.798148 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:40.797868    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:46:45.797719 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:45.797655    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:45.798213 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:45.797971    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:46:45.912204 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:45.912171    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:50.089660 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:50.089628    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:50.797662 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:50.797620    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:55.797291 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:55.797252    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Oct 31 00:46:55.797826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:55.797547    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:46:55.797826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:55.797742    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:46:57.130376 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:57.130332    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:46:57.517657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:57.517625    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:57.661476 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:57.661440    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:58.347385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.347363    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:46:58.429730 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.429709    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:46:58.431768 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.431751    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\" (UniqueName: \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\") pod \"d0319cf0-3b3f-4f06-abda-2441481e376d\" (UID: \"d0319cf0-3b3f-4f06-abda-2441481e376d\") "
Oct 31 00:46:58.433806 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.433775    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd" pod "d0319cf0-3b3f-4f06-abda-2441481e376d" (UID: "d0319cf0-3b3f-4f06-abda-2441481e376d"). InnerVolumeSpecName "kube-api-access-9fdrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:58.532427 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.532401    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\" (UniqueName: \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\") pod \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") "
Oct 31 00:46:58.532581 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.532505    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") pod \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") "
Oct 31 00:46:58.532581 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.532549    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\") pod \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") "
Oct 31 00:46:58.532731 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.532580    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\") pod \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\" (UID: \"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3\") "
Oct 31 00:46:58.532795 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.532772    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdrd\" (UniqueName: \"kubernetes.io/projected/d0319cf0-3b3f-4f06-abda-2441481e376d-kube-api-access-9fdrd\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:58.533045 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.533008    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config" pod "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" (UID: "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:58.534732 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.534703    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k" pod "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" (UID: "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3"). InnerVolumeSpecName "kube-api-access-rlj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:58.534837 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.534785    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls" pod "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" (UID: "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:58.535597 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.535577    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78" pod "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" (UID: "7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3"). InnerVolumeSpecName "pvc-93933fae-9f0d-46de-b846-59ea91b29a09". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:46:58.633429 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.633396    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlj5k\" (UniqueName: \"kubernetes.io/projected/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-api-access-rlj5k\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:58.633535 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.633443    4553 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" "
Oct 31 00:46:58.633535 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.633454    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:58.633535 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.633463    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:58.644930 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.644914    4553 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-93933fae-9f0d-46de-b846-59ea91b29a09" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78") on node "ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:46:58.734662 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:58.734603    4553 reconciler_common.go:299] "Volume detached for volume \"pvc-93933fae-9f0d-46de-b846-59ea91b29a09\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09530eecc3cfb3a78\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"csi-5dbef36f7953750475b64f5362d5846ba76274c305650c6a8bde0b40d1cc7b60\""
Oct 31 00:46:59.048007 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.047927    4553 generic.go:356] "Generic (PLEG): container finished" podID="d0319cf0-3b3f-4f06-abda-2441481e376d" containerID="324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673" exitCode=137
Oct 31 00:46:59.048007 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.047979    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"
Oct 31 00:46:59.048239 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.048007    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" event={"ID":"d0319cf0-3b3f-4f06-abda-2441481e376d","Type":"ContainerDied","Data":"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"}
Oct 31 00:46:59.048239 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.048090    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs" event={"ID":"d0319cf0-3b3f-4f06-abda-2441481e376d","Type":"ContainerDied","Data":"6728c8e61ac40a5d974c235b433b942b83b83053018a487b5b9b574812a44e2e"}
Oct 31 00:46:59.048239 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.048122    4553 scope.go:122] "RemoveContainer" containerID="324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"
Oct 31 00:46:59.050110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050078    4553 generic.go:356] "Generic (PLEG): container finished" podID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerID="4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b" exitCode=0
Oct 31 00:46:59.050110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050106    4553 generic.go:356] "Generic (PLEG): container finished" podID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerID="8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e" exitCode=137
Oct 31 00:46:59.050311 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050106    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerDied","Data":"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"}
Oct 31 00:46:59.050311 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050159    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"
Oct 31 00:46:59.050311 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerDied","Data":"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"}
Oct 31 00:46:59.050311 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.050209    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw" event={"ID":"7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3","Type":"ContainerDied","Data":"194cf466a9b19b65571be84f7af889c81b1b11f0bc7eb501496aac3df5f849c4"}
Oct 31 00:46:59.056228 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.056203    4553 scope.go:122] "RemoveContainer" containerID="324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"
Oct 31 00:46:59.056519 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:59.056500    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673\": container with ID starting with 324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673 not found: ID does not exist" containerID="324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"
Oct 31 00:46:59.056572 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.056530    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673"} err="failed to get container status \"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673\": rpc error: code = NotFound desc = could not find container \"324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673\": container with ID starting with 324780455aaed3f8d9e4a627d328b3d307805a2d3dcdeeda16b751cd5b9dd673 not found: ID does not exist"
Oct 31 00:46:59.056572 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.056551    4553 scope.go:122] "RemoveContainer" containerID="4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"
Oct 31 00:46:59.063956 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.063936    4553 scope.go:122] "RemoveContainer" containerID="d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"
Oct 31 00:46:59.070725 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.070707    4553 scope.go:122] "RemoveContainer" containerID="8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"
Oct 31 00:46:59.077633 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.077614    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:46:59.077705 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.077671    4553 scope.go:122] "RemoveContainer" containerID="4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"
Oct 31 00:46:59.077971 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:59.077955    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b\": container with ID starting with 4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b not found: ID does not exist" containerID="4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"
Oct 31 00:46:59.078018 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.077979    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"} err="failed to get container status \"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b\": rpc error: code = NotFound desc = could not find container \"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b\": container with ID starting with 4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b not found: ID does not exist"
Oct 31 00:46:59.078018 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.077995    4553 scope.go:122] "RemoveContainer" containerID="d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"
Oct 31 00:46:59.078226 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:59.078209    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd\": container with ID starting with d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd not found: ID does not exist" containerID="d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"
Oct 31 00:46:59.078267 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078236    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"} err="failed to get container status \"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd\": rpc error: code = NotFound desc = could not find container \"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd\": container with ID starting with d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd not found: ID does not exist"
Oct 31 00:46:59.078267 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078252    4553 scope.go:122] "RemoveContainer" containerID="8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"
Oct 31 00:46:59.078465 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:46:59.078451    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e\": container with ID starting with 8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e not found: ID does not exist" containerID="8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"
Oct 31 00:46:59.078505 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078469    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"} err="failed to get container status \"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e\": rpc error: code = NotFound desc = could not find container \"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e\": container with ID starting with 8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e not found: ID does not exist"
Oct 31 00:46:59.078505 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078482    4553 scope.go:122] "RemoveContainer" containerID="4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"
Oct 31 00:46:59.078665 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078650    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b"} err="failed to get container status \"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b\": rpc error: code = NotFound desc = could not find container \"4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b\": container with ID starting with 4b895370cb156d3e80d20ecf4246ec0e87d51a8399324b03ca86e2251d18087b not found: ID does not exist"
Oct 31 00:46:59.078731 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078665    4553 scope.go:122] "RemoveContainer" containerID="d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"
Oct 31 00:46:59.078868 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078852    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd"} err="failed to get container status \"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd\": rpc error: code = NotFound desc = could not find container \"d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd\": container with ID starting with d5209c64035236609a7c0fe68e60cd6aab6ad8886e1d7ec70672ae34dca619cd not found: ID does not exist"
Oct 31 00:46:59.078909 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.078867    4553 scope.go:122] "RemoveContainer" containerID="8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"
Oct 31 00:46:59.079059 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.079043    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e"} err="failed to get container status \"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e\": rpc error: code = NotFound desc = could not find container \"8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e\": container with ID starting with 8abc49b967822b8606b8f0e04152b21548d09b25f4f5cdd5b452d1248c353e8e not found: ID does not exist"
Oct 31 00:46:59.083986 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.083967    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-sink-5bf48bbf54-94zfs"]
Oct 31 00:46:59.100946 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.100922    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:46:59.114540 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.114518    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-d2160853-predictor-94566c6b9-wvlpw"]
Oct 31 00:46:59.199654 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.199624    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" path="/var/lib/kubelet/pods/7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3/volumes"
Oct 31 00:46:59.200069 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.200054    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0319cf0-3b3f-4f06-abda-2441481e376d" path="/var/lib/kubelet/pods/d0319cf0-3b3f-4f06-abda-2441481e376d/volumes"
Oct 31 00:46:59.244781 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.244752    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:46:59.384726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:46:59.384626    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:01.230933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.230899    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:01.264668 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264641    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:47:01.264960 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264947    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kserve-container"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264962    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kserve-container"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264977    4553 cpu_manager.go:414] "containerMap: removing container" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kserve-container"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264982    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kserve-container"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.264993    4553 cpu_manager.go:414] "containerMap: removing container" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265001    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265013    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265018    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265025    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265029    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265036    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d0319cf0-3b3f-4f06-abda-2441481e376d" containerName="kserve-container"
Oct 31 00:47:01.265038 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265041    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0319cf0-3b3f-4f06-abda-2441481e376d" containerName="kserve-container"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265099    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265106    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="agent"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265111    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kube-rbac-proxy"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265117    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0319cf0-3b3f-4f06-abda-2441481e376d" containerName="kserve-container"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265124    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ba5b3-ad15-4c46-86a1-c0cf049f98a3" containerName="kserve-container"
Oct 31 00:47:01.265571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.265130    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e9aaed-e22d-49f0-b98f-190e6980c441" containerName="kserve-container"
Oct 31 00:47:01.269901 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.269881    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:01.281116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.278359    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:47:01.281116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.278801    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:47:01.281116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.278957    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:47:01.281116 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.279022    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:47:01.356444 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.356413    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kp2\" (UniqueName: \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\") pod \"proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x\" (UID: \"6827563c-8722-4bbe-a83b-a8954c41761f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:01.367773 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.367737    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:01.457308 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.457273    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kp2\" (UniqueName: \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\") pod \"proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x\" (UID: \"6827563c-8722-4bbe-a83b-a8954c41761f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:01.472580 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.472550    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kp2\" (UniqueName: \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\") pod \"proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x\" (UID: \"6827563c-8722-4bbe-a83b-a8954c41761f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:01.584867 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.584782    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:01.859278 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:01.859097    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:47:01.861815 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:47:01.861784    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6827563c_8722_4bbe_a83b_a8954c41761f.slice/crio-3242a0be9b70767d40fa57313e88e925546501bb53db187b6a4abc227ab7cc53 WatchSource:0}: Error finding container 3242a0be9b70767d40fa57313e88e925546501bb53db187b6a4abc227ab7cc53: Status 404 returned error can't find the container with id 3242a0be9b70767d40fa57313e88e925546501bb53db187b6a4abc227ab7cc53
Oct 31 00:47:02.067093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:02.067064    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" event={"ID":"6827563c-8722-4bbe-a83b-a8954c41761f","Type":"ContainerStarted","Data":"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"}
Oct 31 00:47:02.067237 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:02.067111    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" event={"ID":"6827563c-8722-4bbe-a83b-a8954c41761f","Type":"ContainerStarted","Data":"3242a0be9b70767d40fa57313e88e925546501bb53db187b6a4abc227ab7cc53"}
Oct 31 00:47:02.067312 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:02.067290    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:02.068889 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:02.068862    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/\": dial tcp 10.133.0.29:8080: connect: connection refused"
Oct 31 00:47:02.088516 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:02.088468    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" podStartSLOduration=1.088456524 podStartE2EDuration="1.088456524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:02.08807702 +0000 UTC m=+1084.997079674" watchObservedRunningTime="2026-10-31 00:47:02.088456524 +0000 UTC m=+1084.997459179"
Oct 31 00:47:03.072384 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:03.072351    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:47:03.646188 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:03.646158    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:03.784746 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:03.784709    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:03.921020 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:03.920994    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:05.914243 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:05.914199    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:06.054112 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.054079    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:06.451929 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.451896    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:06.593899 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.593868    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:06.639522 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.639498    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:47:06.643095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.643079    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.649633 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.649611    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c4e5dad3-kube-rbac-proxy-sar-config\""
Oct 31 00:47:06.649633 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.649618    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c4e5dad3-dockercfg-2lkmr\""
Oct 31 00:47:06.649797 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.649692    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c4e5dad3-predictor-serving-cert\""
Oct 31 00:47:06.655238 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.655218    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:47:06.698745 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.698724    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.698849 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.698756    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gpk\" (UniqueName: \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.698892 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.698875    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.774779 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.774714    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:06.799860 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.799834    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.799979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.799899    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.799979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.799931    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gpk\" (UniqueName: \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.800432 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.800407    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.802412 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.802391    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:06.814495 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:06.814471    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gpk\" (UniqueName: \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:09.721308 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:09.721267    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:09.730801 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:09.730775    4553 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:09.822171 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:09.822139    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:09.822310 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:09.822242    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:09.835011 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:09.834988    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") DevicePath \"csi-a05caaf88b74d54a12644eac3897998caf52deeccb327d185a399bcf4aee59b5\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:10.001023 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:10.000991    4553 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/325f567af2d6b785ad0cd85a10a52bcbaed090ee6023f8baee156c1f13d04c72/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:10.015314 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:10.015289    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:10.257020 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:10.256950    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:10.535916 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:10.535871    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:47:10.538469 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:47:10.538439    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026 WatchSource:0}: Error finding container e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026: Status 404 returned error can't find the container with id e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026
Oct 31 00:47:11.102217 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.102180    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerStarted","Data":"d72107fcd7425e7432cbc5e181621017871cb377addf86c9d5c37e39ca05ea47"}
Oct 31 00:47:11.102645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.102242    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerStarted","Data":"49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f"}
Oct 31 00:47:11.102645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.102275    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:11.102645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.102300    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerStarted","Data":"e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026"}
Oct 31 00:47:11.102645 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.102327    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:11.104547 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.104525    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:11.128334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:11.128290    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podStartSLOduration=8.128277409 podStartE2EDuration="8.128277409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:11.127594128 +0000 UTC m=+1094.036596782" watchObservedRunningTime="2026-10-31 00:47:11.128277409 +0000 UTC m=+1094.037280062"
Oct 31 00:47:18.114962 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:18.114936    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:19.152515 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:19.152476    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:19.284616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:19.284577    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:23.727773 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.727741    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:47:23.731177 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.731161    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.743972 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.743944    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:47:23.827071 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.827042    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnbw\" (UniqueName: \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.827195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.827090    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.827195 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.827119    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.827269 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.827214    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.827269 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.827226    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") DevicePath \"csi-a05caaf88b74d54a12644eac3897998caf52deeccb327d185a399bcf4aee59b5\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.831582 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.831565    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") DevicePath \"csi-a05caaf88b74d54a12644eac3897998caf52deeccb327d185a399bcf4aee59b5\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.843938 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.843921    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.927595 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.927570    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.927735 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.927604    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.927735 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.927634    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnbw\" (UniqueName: \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.928227 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.928208    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.930037 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.930014    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:23.942546 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:23.942526    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnbw\" (UniqueName: \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\") pod \"proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:24.047700 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:24.047608    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:24.326436 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:24.326413    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:47:24.328643 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:47:24.328616    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9ab19d_7a4d_42d0_96d5_0beab935378a.slice/crio-9c3f404c6ab5000178908ec7cec2796b79c69c279fa5bd03c92b04707da41ff5 WatchSource:0}: Error finding container 9c3f404c6ab5000178908ec7cec2796b79c69c279fa5bd03c92b04707da41ff5: Status 404 returned error can't find the container with id 9c3f404c6ab5000178908ec7cec2796b79c69c279fa5bd03c92b04707da41ff5
Oct 31 00:47:25.158734 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158695    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerStarted","Data":"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"}
Oct 31 00:47:25.159130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158766    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerStarted","Data":"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"}
Oct 31 00:47:25.159130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158795    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:25.159130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158824    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerStarted","Data":"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"}
Oct 31 00:47:25.159130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158845    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerStarted","Data":"9c3f404c6ab5000178908ec7cec2796b79c69c279fa5bd03c92b04707da41ff5"}
Oct 31 00:47:25.159130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.158868    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:25.161097 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.161076    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:25.188224 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:25.188176    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podStartSLOduration=2.188164542 podStartE2EDuration="2.188164542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:25.185829793 +0000 UTC m=+1108.094832454" watchObservedRunningTime="2026-10-31 00:47:25.188164542 +0000 UTC m=+1108.097167195"
Oct 31 00:47:26.162666 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:26.162637    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:26.164088 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:26.164067    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:27.129973 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:47:27.129934    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:47:33.177305 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:33.177279    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:47:33.242077 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:33.242050    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:47:33.242343 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:33.242320    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kserve-container" containerID="cri-o://49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f" gracePeriod=30
Oct 31 00:47:33.242427 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:33.242379    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" containerID="cri-o://d72107fcd7425e7432cbc5e181621017871cb377addf86c9d5c37e39ca05ea47" gracePeriod=30
Oct 31 00:47:34.199130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:34.199093    4553 generic.go:356] "Generic (PLEG): container finished" podID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerID="d72107fcd7425e7432cbc5e181621017871cb377addf86c9d5c37e39ca05ea47" exitCode=0
Oct 31 00:47:34.199526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:34.199165    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerDied","Data":"d72107fcd7425e7432cbc5e181621017871cb377addf86c9d5c37e39ca05ea47"}
Oct 31 00:47:38.111166 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:38.111126    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:47:43.111150 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:43.111105    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:47:45.973155 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:45.973099    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:47:48.111319 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:48.111282    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:47:48.111790 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:48.111384    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:47:53.111463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:53.111424    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:47:57.129580 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:47:57.129532    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:47:58.111001 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:47:58.110959    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:48:03.111501 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.111458    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 31 00:48:03.316069 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:03.316035    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-conmon-49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:03.316199 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:03.316103    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-conmon-49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293535e6_e1f1_4eaa_9cb1_ae8d727b80f4.slice/crio-49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:48:03.318824 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.318790    4553 generic.go:356] "Generic (PLEG): container finished" podID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerID="49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f" exitCode=137
Oct 31 00:48:03.319049 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.319021    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerDied","Data":"49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f"}
Oct 31 00:48:03.442938 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.442910    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:48:03.555325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.555297    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\") pod \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") "
Oct 31 00:48:03.555493 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.555340    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\") pod \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") "
Oct 31 00:48:03.555493 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.555444    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") "
Oct 31 00:48:03.555493 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.555479    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\" (UniqueName: \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\") pod \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\" (UID: \"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4\") "
Oct 31 00:48:03.555741 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.555717    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config" pod "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" (UID: "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:03.557706 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.557665    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls" pod "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" (UID: "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:03.557831 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.557766    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk" pod "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" (UID: "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4"). InnerVolumeSpecName "kube-api-access-p7gpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:03.558395 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.558375    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7" pod "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" (UID: "293535e6-e1f1-4eaa-9cb1-ae8d727b80f4"). InnerVolumeSpecName "pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:48:03.656128 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.656103    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:03.656128 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.656127    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:03.656255 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:03.656138    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gpk\" (UniqueName: \"kubernetes.io/projected/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4-kube-api-access-p7gpk\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:04.324519 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.324491    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"
Oct 31 00:48:04.324519 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.324498    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f" event={"ID":"293535e6-e1f1-4eaa-9cb1-ae8d727b80f4","Type":"ContainerDied","Data":"e8c496f899eab5dcd75203609e934379e8ab738446c37b4c135fca8667fe7026"}
Oct 31 00:48:04.325217 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.324559    4553 scope.go:122] "RemoveContainer" containerID="d72107fcd7425e7432cbc5e181621017871cb377addf86c9d5c37e39ca05ea47"
Oct 31 00:48:04.333471 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.333454    4553 scope.go:122] "RemoveContainer" containerID="49ce7bc654efb046a84d0f58c7c3da732e45df5c59475505fc4af58838cdbc9f"
Oct 31 00:48:04.350543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.350520    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:48:04.356715 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:04.356696    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-6cdc5469b8-5mn4f"]
Oct 31 00:48:05.199437 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:05.199405    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" path="/var/lib/kubelet/pods/293535e6-e1f1-4eaa-9cb1-ae8d727b80f4/volumes"
Oct 31 00:48:10.353223 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.353193    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-f4cda2dafd_proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq_9d9ab19d-7a4d-42d0-96d5-0beab935378a/kserve-container/0.log"
Oct 31 00:48:10.457600 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.457567    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:48:10.457862 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.457837    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" containerName="kserve-container" containerID="cri-o://670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d" gracePeriod=30
Oct 31 00:48:10.519132 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.519102    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:10.610521 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.610458    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:48:10.610729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.610690    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kserve-container" containerID="cri-o://517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76" gracePeriod=30
Oct 31 00:48:10.610864 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.610732    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent" containerID="cri-o://c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608" gracePeriod=30
Oct 31 00:48:10.610864 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.610729    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" containerID="cri-o://2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c" gracePeriod=30
Oct 31 00:48:10.661438 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.661406    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:10.880005 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:10.879925    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:11.354354 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:11.354324    4553 generic.go:356] "Generic (PLEG): container finished" podID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerID="2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c" exitCode=0
Oct 31 00:48:11.354760 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:11.354375    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerDied","Data":"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"}
Oct 31 00:48:13.173889 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:13.173842    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:16.163206 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:16.163165    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:48:18.174041 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:18.174003    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:18.394078 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:18.394043    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:23.174148 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:23.174098    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:23.174558 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:23.174276    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:48:25.799605 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:25.799566    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:25.949532 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:25.949495    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:26.163951 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:26.163917    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:48:27.129547 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:27.129511    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:48:28.173748 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:28.173714    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:33.174434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:33.174392    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:36.164182 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:36.164141    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:48:36.164700 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:36.164289    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:48:38.174189 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:38.174148    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Oct 31 00:48:40.688077 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.688053    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:48:40.808012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.807988    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:48:40.834922 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.834894    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\" (UniqueName: \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\") pod \"6827563c-8722-4bbe-a83b-a8954c41761f\" (UID: \"6827563c-8722-4bbe-a83b-a8954c41761f\") "
Oct 31 00:48:40.837023 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.837001    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2" pod "6827563c-8722-4bbe-a83b-a8954c41761f" (UID: "6827563c-8722-4bbe-a83b-a8954c41761f"). InnerVolumeSpecName "kube-api-access-d5kp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:40.936084 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936041    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\") pod \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") "
Oct 31 00:48:40.936263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936181    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") pod \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") "
Oct 31 00:48:40.936263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936225    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\") pod \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") "
Oct 31 00:48:40.936340 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936278    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\" (UniqueName: \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\") pod \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\" (UID: \"9d9ab19d-7a4d-42d0-96d5-0beab935378a\") "
Oct 31 00:48:40.936410 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936386    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config" pod "9d9ab19d-7a4d-42d0-96d5-0beab935378a" (UID: "9d9ab19d-7a4d-42d0-96d5-0beab935378a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:40.936533 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936516    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kp2\" (UniqueName: \"kubernetes.io/projected/6827563c-8722-4bbe-a83b-a8954c41761f-kube-api-access-d5kp2\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:40.936567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.936541    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:40.938518 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.938487    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls" pod "9d9ab19d-7a4d-42d0-96d5-0beab935378a" (UID: "9d9ab19d-7a4d-42d0-96d5-0beab935378a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:40.938620 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.938556    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw" pod "9d9ab19d-7a4d-42d0-96d5-0beab935378a" (UID: "9d9ab19d-7a4d-42d0-96d5-0beab935378a"). InnerVolumeSpecName "kube-api-access-qvnbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:40.939025 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:40.939005    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7" pod "9d9ab19d-7a4d-42d0-96d5-0beab935378a" (UID: "9d9ab19d-7a4d-42d0-96d5-0beab935378a"). InnerVolumeSpecName "pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:48:41.037124 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.037017    4553 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" "
Oct 31 00:48:41.037124 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.037065    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d9ab19d-7a4d-42d0-96d5-0beab935378a-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:41.037124 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.037081    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnbw\" (UniqueName: \"kubernetes.io/projected/9d9ab19d-7a4d-42d0-96d5-0beab935378a-kube-api-access-qvnbw\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:41.046330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.046312    4553 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7") on node "ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:48:41.138010 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.137964    4553 reconciler_common.go:299] "Volume detached for volume \"pvc-7ab893d1-371e-479b-8f5e-92c1b5a85f8a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09ca57cbb3b7b8bb7\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"csi-a05caaf88b74d54a12644eac3897998caf52deeccb327d185a399bcf4aee59b5\""
Oct 31 00:48:41.470185 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470149    4553 generic.go:356] "Generic (PLEG): container finished" podID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerID="c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608" exitCode=0
Oct 31 00:48:41.470185 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470181    4553 generic.go:356] "Generic (PLEG): container finished" podID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerID="517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76" exitCode=137
Oct 31 00:48:41.470409 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470251    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"
Oct 31 00:48:41.470409 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470261    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerDied","Data":"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"}
Oct 31 00:48:41.470409 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470330    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerDied","Data":"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"}
Oct 31 00:48:41.470409 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470365    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq" event={"ID":"9d9ab19d-7a4d-42d0-96d5-0beab935378a","Type":"ContainerDied","Data":"9c3f404c6ab5000178908ec7cec2796b79c69c279fa5bd03c92b04707da41ff5"}
Oct 31 00:48:41.470409 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.470403    4553 scope.go:122] "RemoveContainer" containerID="c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"
Oct 31 00:48:41.471888 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.471564    4553 generic.go:356] "Generic (PLEG): container finished" podID="6827563c-8722-4bbe-a83b-a8954c41761f" containerID="670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d" exitCode=137
Oct 31 00:48:41.471888 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.471604    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"
Oct 31 00:48:41.471888 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.471644    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" event={"ID":"6827563c-8722-4bbe-a83b-a8954c41761f","Type":"ContainerDied","Data":"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"}
Oct 31 00:48:41.471888 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.471707    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x" event={"ID":"6827563c-8722-4bbe-a83b-a8954c41761f","Type":"ContainerDied","Data":"3242a0be9b70767d40fa57313e88e925546501bb53db187b6a4abc227ab7cc53"}
Oct 31 00:48:41.479730 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.479714    4553 scope.go:122] "RemoveContainer" containerID="2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"
Oct 31 00:48:41.487012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.486994    4553 scope.go:122] "RemoveContainer" containerID="517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"
Oct 31 00:48:41.494156 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494134    4553 scope.go:122] "RemoveContainer" containerID="c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"
Oct 31 00:48:41.494423 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:41.494405    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608\": container with ID starting with c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608 not found: ID does not exist" containerID="c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"
Oct 31 00:48:41.494473 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494430    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"} err="failed to get container status \"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608\": rpc error: code = NotFound desc = could not find container \"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608\": container with ID starting with c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608 not found: ID does not exist"
Oct 31 00:48:41.494473 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494446    4553 scope.go:122] "RemoveContainer" containerID="2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"
Oct 31 00:48:41.494708 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:41.494686    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c\": container with ID starting with 2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c not found: ID does not exist" containerID="2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"
Oct 31 00:48:41.494791 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494712    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"} err="failed to get container status \"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c\": rpc error: code = NotFound desc = could not find container \"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c\": container with ID starting with 2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c not found: ID does not exist"
Oct 31 00:48:41.494791 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494727    4553 scope.go:122] "RemoveContainer" containerID="517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"
Oct 31 00:48:41.494975 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:41.494958    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76\": container with ID starting with 517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76 not found: ID does not exist" containerID="517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"
Oct 31 00:48:41.495018 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494983    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"} err="failed to get container status \"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76\": rpc error: code = NotFound desc = could not find container \"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76\": container with ID starting with 517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76 not found: ID does not exist"
Oct 31 00:48:41.495018 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.494997    4553 scope.go:122] "RemoveContainer" containerID="c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"
Oct 31 00:48:41.495244 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495225    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608"} err="failed to get container status \"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608\": rpc error: code = NotFound desc = could not find container \"c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608\": container with ID starting with c3c72873393ce38706f16d1e5a960a32e4892d0bbd7edab9f4829302e962f608 not found: ID does not exist"
Oct 31 00:48:41.495313 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495246    4553 scope.go:122] "RemoveContainer" containerID="2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"
Oct 31 00:48:41.495487 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495471    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c"} err="failed to get container status \"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c\": rpc error: code = NotFound desc = could not find container \"2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c\": container with ID starting with 2221b89f19ebcf381af44fa5ab483b7615b88c387853cd60f4b05875c1c84c8c not found: ID does not exist"
Oct 31 00:48:41.495533 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495488    4553 scope.go:122] "RemoveContainer" containerID="517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"
Oct 31 00:48:41.495708 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495666    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76"} err="failed to get container status \"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76\": rpc error: code = NotFound desc = could not find container \"517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76\": container with ID starting with 517cb3d4577673f07d4815cbe271c8ad6a6a5378fe090397bb1152accae11a76 not found: ID does not exist"
Oct 31 00:48:41.495790 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.495710    4553 scope.go:122] "RemoveContainer" containerID="670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"
Oct 31 00:48:41.498941 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.498921    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:48:41.503033 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.503019    4553 scope.go:122] "RemoveContainer" containerID="670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"
Oct 31 00:48:41.503275 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:41.503256    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d\": container with ID starting with 670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d not found: ID does not exist" containerID="670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"
Oct 31 00:48:41.503341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.503280    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d"} err="failed to get container status \"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d\": rpc error: code = NotFound desc = could not find container \"670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d\": container with ID starting with 670ee12afd3799cb90adb2b7b05fbb0e8ea6df20b9345238aa4fd01065dc103d not found: ID does not exist"
Oct 31 00:48:41.505413 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.505394    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-predictor-68c47c8f5b-tw6pq"]
Oct 31 00:48:41.521297 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.521279    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:48:41.527475 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.527453    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c4e5dad3-sink-ff979df7b-p2p6x"]
Oct 31 00:48:41.644012 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.643958    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:41.800948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:41.800869    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:43.199933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.199898    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" path="/var/lib/kubelet/pods/6827563c-8722-4bbe-a83b-a8954c41761f/volumes"
Oct 31 00:48:43.200327 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.200273    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" path="/var/lib/kubelet/pods/9d9ab19d-7a4d-42d0-96d5-0beab935378a/volumes"
Oct 31 00:48:43.514065 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.513982    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:43.532540 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.532506    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:48:43.532980 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.532957    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" containerName="kserve-container"
Oct 31 00:48:43.533176 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533152    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" containerName="kserve-container"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533183    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kserve-container"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533192    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kserve-container"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533207    4553 cpu_manager.go:414] "containerMap: removing container" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kserve-container"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533215    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kserve-container"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533232    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent"
Oct 31 00:48:43.533263 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533239    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533265    4553 cpu_manager.go:414] "containerMap: removing container" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533273    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533283    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533290    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533398    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kserve-container"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533409    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="agent"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533421    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="293535e6-e1f1-4eaa-9cb1-ae8d727b80f4" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533428    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kube-rbac-proxy"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533438    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6827563c-8722-4bbe-a83b-a8954c41761f" containerName="kserve-container"
Oct 31 00:48:43.533500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.533446    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ab19d-7a4d-42d0-96d5-0beab935378a" containerName="kserve-container"
Oct 31 00:48:43.537825 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.537808    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:43.545378 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.545356    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:48:43.545785 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.545765    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:48:43.545859 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.545793    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:48:43.549076 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.549054    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:48:43.656085 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.656049    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbjx\" (UniqueName: \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\") pod \"proxy-routing-c8dac472-sink-77f5868f95-255sv\" (UID: \"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:43.661744 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.661703    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:43.757107 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.757067    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbjx\" (UniqueName: \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\") pod \"proxy-routing-c8dac472-sink-77f5868f95-255sv\" (UID: \"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:43.772486 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.772425    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbjx\" (UniqueName: \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\") pod \"proxy-routing-c8dac472-sink-77f5868f95-255sv\" (UID: \"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:43.849425 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:43.849393    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:44.122658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.122478    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:48:44.125309 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:48:44.125281    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffb5e42_2de2_4d60_9eb0_51f5ff8fa353.slice/crio-294d33e0476b1ac0e11609f6d636d98ebe78e5347584e025b578ce4b73a81395 WatchSource:0}: Error finding container 294d33e0476b1ac0e11609f6d636d98ebe78e5347584e025b578ce4b73a81395: Status 404 returned error can't find the container with id 294d33e0476b1ac0e11609f6d636d98ebe78e5347584e025b578ce4b73a81395
Oct 31 00:48:44.487689 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.487653    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" event={"ID":"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353","Type":"ContainerStarted","Data":"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"}
Oct 31 00:48:44.488173 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.487728    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" event={"ID":"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353","Type":"ContainerStarted","Data":"294d33e0476b1ac0e11609f6d636d98ebe78e5347584e025b578ce4b73a81395"}
Oct 31 00:48:44.488173 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.487756    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:44.490021 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.490002    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:48:44.509334 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.509291    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" podStartSLOduration=1.5092794330000001 podStartE2EDuration="1.509279433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:44.508930639 +0000 UTC m=+1187.417933292" watchObservedRunningTime="2026-10-31 00:48:44.509279433 +0000 UTC m=+1187.418282087"
Oct 31 00:48:44.813995 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.813908    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:44.951111 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:44.951075    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:45.087875 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:45.087792    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:46.040506 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:46.040456    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:46.189343 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:46.189309    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:47.198647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.198607    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:47.346152 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.346115    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:47.919119 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.919088    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:48:47.922447 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.922428    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:47.928735 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.928716    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c8dac472-dockercfg-nv6rl\""
Oct 31 00:48:47.929628 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.929607    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c8dac472-kube-rbac-proxy-sar-config\""
Oct 31 00:48:47.929760 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.929710    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-c8dac472-predictor-serving-cert\""
Oct 31 00:48:47.934910 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:47.934889    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:48:48.093855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.093815    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw46m\" (UniqueName: \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.093855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.093856    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.094090 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.093967    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.134293 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.134256    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:48.195139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.195055    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.195139 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.195111    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw46m\" (UniqueName: \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.195350 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.195156    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.195776 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.195755    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.197647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.197630    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:48.210340 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:48.210313    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw46m\" (UniqueName: \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:49.490980 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:49.490945    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:49.654629 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:49.654595    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:48:51.521287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.521256    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.530571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.530542    4553 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.621692 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.621645    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.621846 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.621796    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.626445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.626427    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") DevicePath \"csi-9e0cc33f2635bd94f6792721f94bf5b0ce9be7a3800f1508a1d3a6ba44e268a5\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.793216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.793189    4553 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/83fc75d4df25273404c55d0eb38b8f78a961ad048b8fa216e22a79fab4819fb5/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.807794 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.807765    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:51.835653 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:51.835616    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:52.112580 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.112556    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:48:52.115006 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:48:52.114971    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac557c70_c889_4d2e_b2d8_b6a202513f96.slice/crio-369dd6acd8d4c253b945fe8007495060ccaebabed6378dceb646afce09c15e96 WatchSource:0}: Error finding container 369dd6acd8d4c253b945fe8007495060ccaebabed6378dceb646afce09c15e96: Status 404 returned error can't find the container with id 369dd6acd8d4c253b945fe8007495060ccaebabed6378dceb646afce09c15e96
Oct 31 00:48:52.518959 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.518925    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerStarted","Data":"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"}
Oct 31 00:48:52.519122 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.518978    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerStarted","Data":"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"}
Oct 31 00:48:52.519122 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.519001    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:52.519122 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.519018    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerStarted","Data":"369dd6acd8d4c253b945fe8007495060ccaebabed6378dceb646afce09c15e96"}
Oct 31 00:48:52.519122 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.519034    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:52.521098 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.521076    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:48:52.545086 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:52.545044    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podStartSLOduration=8.545032836 podStartE2EDuration="8.545032836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:52.543839316 +0000 UTC m=+1195.452841970" watchObservedRunningTime="2026-10-31 00:48:52.545032836 +0000 UTC m=+1195.454035489"
Oct 31 00:48:57.128592 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:48:57.128554    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:48:57.277647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:57.277619    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:48:57.279663 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:57.279641    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:48:59.531250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:48:59.531223    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:49:03.887378 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:03.887337    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:49:27.129722 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:49:27.129615    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:49:35.914698 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:35.914651    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:49:35.915170 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:35.914871    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" podUID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" containerName="kserve-container" containerID="cri-o://55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9" gracePeriod=30
Oct 31 00:49:35.995769 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:35.995735    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:49:36.037768 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:36.037742    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:49:36.038029 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:36.038002    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kserve-container" containerID="cri-o://2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462" gracePeriod=30
Oct 31 00:49:36.038142 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:36.038022    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" containerID="cri-o://a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b" gracePeriod=30
Oct 31 00:49:36.672231 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:36.672195    4553 generic.go:356] "Generic (PLEG): container finished" podID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerID="a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b" exitCode=0
Oct 31 00:49:36.672397 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:36.672263    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerDied","Data":"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"}
Oct 31 00:49:37.832669 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:37.832621    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:49:39.527848 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:39.527799    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:49:44.527586 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:44.527547    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:49:46.091462 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:46.091419    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:49:49.526932 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:49.526891    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:49:49.527412 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:49.527003    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:49:54.527283 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:54.527246    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:49:57.129348 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:49:57.129310    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:49:59.527887 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:49:59.527841    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:50:04.527580 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:04.527534    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 31 00:50:06.118475 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.118454    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:50:06.166066 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.166041    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\" (UniqueName: \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\") pod \"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353\" (UID: \"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353\") "
Oct 31 00:50:06.168823 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.168761    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx" pod "bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" (UID: "bffb5e42-2de2-4d60-9eb0-51f5ff8fa353"). InnerVolumeSpecName "kube-api-access-zqbjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:06.228951 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.228931    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:50:06.267435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267413    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\") pod \"ac557c70-c889-4d2e-b2d8-b6a202513f96\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") "
Oct 31 00:50:06.267559 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267544    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") pod \"ac557c70-c889-4d2e-b2d8-b6a202513f96\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") "
Oct 31 00:50:06.267701 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267603    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\") pod \"ac557c70-c889-4d2e-b2d8-b6a202513f96\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") "
Oct 31 00:50:06.267768 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267663    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\" (UniqueName: \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\") pod \"ac557c70-c889-4d2e-b2d8-b6a202513f96\" (UID: \"ac557c70-c889-4d2e-b2d8-b6a202513f96\") "
Oct 31 00:50:06.267768 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267732    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config" pod "ac557c70-c889-4d2e-b2d8-b6a202513f96" (UID: "ac557c70-c889-4d2e-b2d8-b6a202513f96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:06.268216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.267984    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbjx\" (UniqueName: \"kubernetes.io/projected/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353-kube-api-access-zqbjx\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:06.268216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.268008    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:06.269869 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.269845    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls" pod "ac557c70-c889-4d2e-b2d8-b6a202513f96" (UID: "ac557c70-c889-4d2e-b2d8-b6a202513f96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:06.270337 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.270309    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m" pod "ac557c70-c889-4d2e-b2d8-b6a202513f96" (UID: "ac557c70-c889-4d2e-b2d8-b6a202513f96"). InnerVolumeSpecName "kube-api-access-mw46m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:06.270431 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.270365    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e" pod "ac557c70-c889-4d2e-b2d8-b6a202513f96" (UID: "ac557c70-c889-4d2e-b2d8-b6a202513f96"). InnerVolumeSpecName "pvc-e7ff20b7-b03d-407a-beef-92240267bc9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:50:06.368570 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.368547    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw46m\" (UniqueName: \"kubernetes.io/projected/ac557c70-c889-4d2e-b2d8-b6a202513f96-kube-api-access-mw46m\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:06.368709 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.368587    4553 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" "
Oct 31 00:50:06.368709 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.368602    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac557c70-c889-4d2e-b2d8-b6a202513f96-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:06.377341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.377326    4553 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7ff20b7-b03d-407a-beef-92240267bc9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e") on node "ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:50:06.469745 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.469698    4553 reconciler_common.go:299] "Volume detached for volume \"pvc-e7ff20b7-b03d-407a-beef-92240267bc9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b5160e6e97161b4e\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"csi-9e0cc33f2635bd94f6792721f94bf5b0ce9be7a3800f1508a1d3a6ba44e268a5\""
Oct 31 00:50:06.777465 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.777379    4553 generic.go:356] "Generic (PLEG): container finished" podID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" containerID="55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9" exitCode=137
Oct 31 00:50:06.777465 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.777426    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"
Oct 31 00:50:06.777717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.777464    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" event={"ID":"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353","Type":"ContainerDied","Data":"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"}
Oct 31 00:50:06.777717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.777510    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv" event={"ID":"bffb5e42-2de2-4d60-9eb0-51f5ff8fa353","Type":"ContainerDied","Data":"294d33e0476b1ac0e11609f6d636d98ebe78e5347584e025b578ce4b73a81395"}
Oct 31 00:50:06.777717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.777535    4553 scope.go:122] "RemoveContainer" containerID="55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"
Oct 31 00:50:06.779215 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.779191    4553 generic.go:356] "Generic (PLEG): container finished" podID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerID="2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462" exitCode=137
Oct 31 00:50:06.779325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.779237    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"
Oct 31 00:50:06.779325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.779251    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerDied","Data":"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"}
Oct 31 00:50:06.779325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.779301    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq" event={"ID":"ac557c70-c889-4d2e-b2d8-b6a202513f96","Type":"ContainerDied","Data":"369dd6acd8d4c253b945fe8007495060ccaebabed6378dceb646afce09c15e96"}
Oct 31 00:50:06.786999 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.786978    4553 scope.go:122] "RemoveContainer" containerID="55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"
Oct 31 00:50:06.787277 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:50:06.787260    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9\": container with ID starting with 55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9 not found: ID does not exist" containerID="55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"
Oct 31 00:50:06.787353 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.787288    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9"} err="failed to get container status \"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9\": rpc error: code = NotFound desc = could not find container \"55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9\": container with ID starting with 55e31727b5e0379393fbb40f70769eb4613177da9de9fb7e6fdf8756178ed1e9 not found: ID does not exist"
Oct 31 00:50:06.787353 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.787309    4553 scope.go:122] "RemoveContainer" containerID="a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"
Oct 31 00:50:06.794773 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.794754    4553 scope.go:122] "RemoveContainer" containerID="2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"
Oct 31 00:50:06.801625 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.801609    4553 scope.go:122] "RemoveContainer" containerID="a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"
Oct 31 00:50:06.801961 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:50:06.801943    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b\": container with ID starting with a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b not found: ID does not exist" containerID="a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"
Oct 31 00:50:06.802013 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.801968    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b"} err="failed to get container status \"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b\": rpc error: code = NotFound desc = could not find container \"a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b\": container with ID starting with a5a6bc798e0d708ec017e037a9f8797884998a93245f8aafb819a8135112627b not found: ID does not exist"
Oct 31 00:50:06.802013 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.801984    4553 scope.go:122] "RemoveContainer" containerID="2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"
Oct 31 00:50:06.802217 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:50:06.802201    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462\": container with ID starting with 2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462 not found: ID does not exist" containerID="2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"
Oct 31 00:50:06.802256 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.802222    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462"} err="failed to get container status \"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462\": rpc error: code = NotFound desc = could not find container \"2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462\": container with ID starting with 2878523e22cce1e60cdcdbd8e644f43dd6978632559f6358f8402d70a5483462 not found: ID does not exist"
Oct 31 00:50:06.804152 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.804132    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:50:06.810146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.810127    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-sink-77f5868f95-255sv"]
Oct 31 00:50:06.826041 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.826022    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:50:06.832307 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.832289    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-c8dac472-predictor-69c7f7bc4c-pjprq"]
Oct 31 00:50:06.960990 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:06.960964    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:07.111169 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:07.111081    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:07.199483 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:07.199445    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" path="/var/lib/kubelet/pods/ac557c70-c889-4d2e-b2d8-b6a202513f96/volumes"
Oct 31 00:50:07.199985 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:07.199963    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" path="/var/lib/kubelet/pods/bffb5e42-2de2-4d60-9eb0-51f5ff8fa353/volumes"
Oct 31 00:50:09.045442 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045411    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045717    4553 cpu_manager.go:414] "containerMap: removing container" podUID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045729    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045743    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045748    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045764    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045769    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045818    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffb5e42-2de2-4d60-9eb0-51f5ff8fa353" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045826    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kserve-container"
Oct 31 00:50:09.045931 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.045835    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac557c70-c889-4d2e-b2d8-b6a202513f96" containerName="kube-rbac-proxy"
Oct 31 00:50:09.050210 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.050184    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.050345 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.050310    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:09.059710 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.058992    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:50:09.059710 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.059195    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:50:09.059710 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.059233    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:50:09.059710 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.059233    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:50:09.181778 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.181744    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:09.188793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.188763    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbjk\" (UniqueName: \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\") pod \"proxy-routing-b388b031-sink-586698bc5d-c4g6b\" (UID: \"44c29a56-2eef-466a-aa63-3ae8d2a77067\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.289760 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.289730    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbjk\" (UniqueName: \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\") pod \"proxy-routing-b388b031-sink-586698bc5d-c4g6b\" (UID: \"44c29a56-2eef-466a-aa63-3ae8d2a77067\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.304702 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.304622    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbjk\" (UniqueName: \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\") pod \"proxy-routing-b388b031-sink-586698bc5d-c4g6b\" (UID: \"44c29a56-2eef-466a-aa63-3ae8d2a77067\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.363565 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.363541    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.635327 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.635251    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:50:09.638110 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:50:09.638080    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c29a56_2eef_466a_aa63_3ae8d2a77067.slice/crio-c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f WatchSource:0}: Error finding container c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f: Status 404 returned error can't find the container with id c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f
Oct 31 00:50:09.798552 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.798514    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" event={"ID":"44c29a56-2eef-466a-aa63-3ae8d2a77067","Type":"ContainerStarted","Data":"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"}
Oct 31 00:50:09.798753 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.798569    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" event={"ID":"44c29a56-2eef-466a-aa63-3ae8d2a77067","Type":"ContainerStarted","Data":"c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f"}
Oct 31 00:50:09.798753 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.798667    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:09.799952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.799922    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/\": dial tcp 10.133.0.34:8080: connect: connection refused"
Oct 31 00:50:09.818874 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:09.818835    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" podStartSLOduration=0.81882347 podStartE2EDuration="818.82347ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:09.81857746 +0000 UTC m=+1272.727580114" watchObservedRunningTime="2026-10-31 00:50:09.81882347 +0000 UTC m=+1272.727826124"
Oct 31 00:50:10.029145 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:10.029110    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:10.804289 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:10.804265    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:50:11.447442 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:11.447409    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:11.585289 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:11.585257    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:11.729335 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:11.729265    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:13.790933 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:13.790896    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:13.930245 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:13.930196    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:14.509493 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.509463    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:50:14.512887 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.512870    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.518730 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.518703    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-b388b031-kube-rbac-proxy-sar-config\""
Oct 31 00:50:14.519479 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.519458    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-b388b031-dockercfg-bjzfw\""
Oct 31 00:50:14.519567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.519529    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-b388b031-predictor-serving-cert\""
Oct 31 00:50:14.526890 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.526871    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:50:14.630376 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.630347    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.630536 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.630381    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.630536 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.630414    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hhz\" (UniqueName: \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.671792 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.671760    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:14.731160 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.731133    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.731283 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.731168    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.731283 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.731200    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hhz\" (UniqueName: \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.731800 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.731781    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.733774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.733755    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:14.745979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:14.745953    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hhz\" (UniqueName: \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:15.319315 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:15.319278    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:15.490592 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:15.490557    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:17.698283 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:17.698244    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:22.148836 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:22.148798    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:22.297118 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:22.297079    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:23.295052 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.295023    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.304322 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.304301    4553 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.396237 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.396203    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.396462 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.396359    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.400982 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.400962    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") DevicePath \"csi-9dc8aec39df6a9034c3b91ef5b9e21ec0048ecf75681182e2e50e35b46b2bf9c\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.569331 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.569303    4553 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1b4938bde24e139342e917d7150e5a52b0af7762c04cfb3f19c8052446fa7307/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.585510 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.585478    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"proxy-routing-b388b031-predictor-6b6fb98846-cf2v9\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:23.828272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:23.828199    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.106100 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.106026    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:50:24.109132 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:50:24.109108    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe822a_d185_46d0_b899_87c01749f371.slice/crio-d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b WatchSource:0}: Error finding container d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b: Status 404 returned error can't find the container with id d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b
Oct 31 00:50:24.856298 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856259    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerStarted","Data":"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"}
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856317    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerStarted","Data":"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"}
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856341    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerStarted","Data":"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"}
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856362    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerStarted","Data":"d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b"}
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856479    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856543    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.856802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.856567    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.859004 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.858983    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.859445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.859426    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:24.886463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:24.886418    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podStartSLOduration=13.886406227 podStartE2EDuration="13.886406227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:24.884052969 +0000 UTC m=+1287.793055623" watchObservedRunningTime="2026-10-31 00:50:24.886406227 +0000 UTC m=+1287.795408885"
Oct 31 00:50:27.129663 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:50:27.129626    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:50:31.869484 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:31.869451    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:50:46.153190 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:50:46.153106    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:50:57.129238 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:50:57.129198    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:51:11.649809 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.649769    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:51:11.650251 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.649969    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerName="kserve-container" containerID="cri-o://e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055" gracePeriod=30
Oct 31 00:51:11.703778 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.703749    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:11.751726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.751701    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:51:11.751986 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.751950    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kserve-container" containerID="cri-o://dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684" gracePeriod=30
Oct 31 00:51:11.752115 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.751987    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent" containerID="cri-o://8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c" gracePeriod=30
Oct 31 00:51:11.752115 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.751987    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" containerID="cri-o://f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7" gracePeriod=30
Oct 31 00:51:11.846432 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.846400    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:11.865720 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:11.865692    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:12.031002 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:12.030975    4553 generic.go:356] "Generic (PLEG): container finished" podID="4efe822a-d185-46d0-b899-87c01749f371" containerID="f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7" exitCode=0
Oct 31 00:51:12.031130 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:12.031013    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerDied","Data":"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"}
Oct 31 00:51:14.857318 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:14.857273    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:51:16.864883 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:16.864844    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:21.865543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:21.865501    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:21.866032 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:21.865623    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:51:24.857182 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:24.857139    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:51:26.865568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:26.865521    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:27.129284 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:27.129197    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:51:31.865041 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:31.864996    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:34.857128 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:34.857078    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:51:34.857573 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:34.857240    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:51:35.750064 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:35.750025    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:36.865192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:36.865150    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 31 00:51:41.840558 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.840529    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:51:41.868793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.868762    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\" (UniqueName: \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\") pod \"44c29a56-2eef-466a-aa63-3ae8d2a77067\" (UID: \"44c29a56-2eef-466a-aa63-3ae8d2a77067\") "
Oct 31 00:51:41.871105 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.871073    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk" pod "44c29a56-2eef-466a-aa63-3ae8d2a77067" (UID: "44c29a56-2eef-466a-aa63-3ae8d2a77067"). InnerVolumeSpecName "kube-api-access-skbjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:41.929227 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.929206    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:51:41.969891 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.969868    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") pod \"4efe822a-d185-46d0-b899-87c01749f371\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") "
Oct 31 00:51:41.970036 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.969902    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\" (UniqueName: \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\") pod \"4efe822a-d185-46d0-b899-87c01749f371\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") "
Oct 31 00:51:41.970036 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.969936    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\") pod \"4efe822a-d185-46d0-b899-87c01749f371\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") "
Oct 31 00:51:41.970036 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.969963    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\") pod \"4efe822a-d185-46d0-b899-87c01749f371\" (UID: \"4efe822a-d185-46d0-b899-87c01749f371\") "
Oct 31 00:51:41.970210 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.970107    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbjk\" (UniqueName: \"kubernetes.io/projected/44c29a56-2eef-466a-aa63-3ae8d2a77067-kube-api-access-skbjk\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:41.970391 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.970349    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config" pod "4efe822a-d185-46d0-b899-87c01749f371" (UID: "4efe822a-d185-46d0-b899-87c01749f371"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:41.972355 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.972326    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls" pod "4efe822a-d185-46d0-b899-87c01749f371" (UID: "4efe822a-d185-46d0-b899-87c01749f371"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:51:41.972467 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.972361    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz" pod "4efe822a-d185-46d0-b899-87c01749f371" (UID: "4efe822a-d185-46d0-b899-87c01749f371"). InnerVolumeSpecName "kube-api-access-l9hhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:41.972921 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:41.972900    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056" pod "4efe822a-d185-46d0-b899-87c01749f371" (UID: "4efe822a-d185-46d0-b899-87c01749f371"). InnerVolumeSpecName "pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:51:42.071176 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.071139    4553 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" "
Oct 31 00:51:42.071176 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.071173    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hhz\" (UniqueName: \"kubernetes.io/projected/4efe822a-d185-46d0-b899-87c01749f371-kube-api-access-l9hhz\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:42.071176 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.071182    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4efe822a-d185-46d0-b899-87c01749f371-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:42.071413 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.071192    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4efe822a-d185-46d0-b899-87c01749f371-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:42.080659 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.080640    4553 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056") on node "ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:51:42.142238 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.142203    4553 generic.go:356] "Generic (PLEG): container finished" podID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerID="e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055" exitCode=137
Oct 31 00:51:42.142390 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.142249    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"
Oct 31 00:51:42.142390 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.142292    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" event={"ID":"44c29a56-2eef-466a-aa63-3ae8d2a77067","Type":"ContainerDied","Data":"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"}
Oct 31 00:51:42.142390 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.142341    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b" event={"ID":"44c29a56-2eef-466a-aa63-3ae8d2a77067","Type":"ContainerDied","Data":"c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f"}
Oct 31 00:51:42.142390 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.142362    4553 scope.go:122] "RemoveContainer" containerID="e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"
Oct 31 00:51:42.144541 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144515    4553 generic.go:356] "Generic (PLEG): container finished" podID="4efe822a-d185-46d0-b899-87c01749f371" containerID="8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c" exitCode=137
Oct 31 00:51:42.144541 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144538    4553 generic.go:356] "Generic (PLEG): container finished" podID="4efe822a-d185-46d0-b899-87c01749f371" containerID="dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684" exitCode=137
Oct 31 00:51:42.144738 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144560    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerDied","Data":"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"}
Oct 31 00:51:42.144738 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144583    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"
Oct 31 00:51:42.144738 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144613    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerDied","Data":"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"}
Oct 31 00:51:42.144738 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.144648    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" event={"ID":"4efe822a-d185-46d0-b899-87c01749f371","Type":"ContainerDied","Data":"d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b"}
Oct 31 00:51:42.158066 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.158038    4553 scope.go:122] "RemoveContainer" containerID="e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"
Oct 31 00:51:42.158381 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:42.158356    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055\": container with ID starting with e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055 not found: ID does not exist" containerID="e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"
Oct 31 00:51:42.158504 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.158393    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055"} err="failed to get container status \"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055\": rpc error: code = NotFound desc = could not find container \"e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055\": container with ID starting with e3fee9f03ceafde9b89eb7a56755af2a0fc12b8f64de44b41b3ff91da1327055 not found: ID does not exist"
Oct 31 00:51:42.158504 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.158418    4553 scope.go:122] "RemoveContainer" containerID="8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"
Oct 31 00:51:42.166105 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.166088    4553 scope.go:122] "RemoveContainer" containerID="f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"
Oct 31 00:51:42.171574 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.171522    4553 reconciler_common.go:299] "Volume detached for volume \"pvc-9200435b-7a52-4157-8e0e-cf81c002a4e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07c2d677b26ca1056\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"csi-9dc8aec39df6a9034c3b91ef5b9e21ec0048ecf75681182e2e50e35b46b2bf9c\""
Oct 31 00:51:42.173949 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.173868    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:51:42.174035 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.174013    4553 scope.go:122] "RemoveContainer" containerID="dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"
Oct 31 00:51:42.179525 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.179504    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-sink-586698bc5d-c4g6b"]
Oct 31 00:51:42.181768 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.181748    4553 scope.go:122] "RemoveContainer" containerID="8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"
Oct 31 00:51:42.182076 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:42.182057    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c\": container with ID starting with 8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c not found: ID does not exist" containerID="8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"
Oct 31 00:51:42.182127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182085    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"} err="failed to get container status \"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c\": rpc error: code = NotFound desc = could not find container \"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c\": container with ID starting with 8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c not found: ID does not exist"
Oct 31 00:51:42.182127 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182102    4553 scope.go:122] "RemoveContainer" containerID="f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"
Oct 31 00:51:42.182361 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:42.182343    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7\": container with ID starting with f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7 not found: ID does not exist" containerID="f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"
Oct 31 00:51:42.182405 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182368    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"} err="failed to get container status \"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7\": rpc error: code = NotFound desc = could not find container \"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7\": container with ID starting with f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7 not found: ID does not exist"
Oct 31 00:51:42.182405 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182385    4553 scope.go:122] "RemoveContainer" containerID="dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"
Oct 31 00:51:42.182632 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:42.182616    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684\": container with ID starting with dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684 not found: ID does not exist" containerID="dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"
Oct 31 00:51:42.182702 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182636    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"} err="failed to get container status \"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684\": rpc error: code = NotFound desc = could not find container \"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684\": container with ID starting with dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684 not found: ID does not exist"
Oct 31 00:51:42.182702 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182650    4553 scope.go:122] "RemoveContainer" containerID="8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"
Oct 31 00:51:42.182921 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182904    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c"} err="failed to get container status \"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c\": rpc error: code = NotFound desc = could not find container \"8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c\": container with ID starting with 8bab4522af15b7b96dbb9a281a1a0aeaf3c1e30ab1ca53166651613c8f4b036c not found: ID does not exist"
Oct 31 00:51:42.182968 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.182922    4553 scope.go:122] "RemoveContainer" containerID="f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"
Oct 31 00:51:42.182968 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:42.182924    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe822a_d185_46d0_b899_87c01749f371.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c29a56_2eef_466a_aa63_3ae8d2a77067.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c29a56_2eef_466a_aa63_3ae8d2a77067.slice/crio-c4d1383dcb14c72bceefeaf2fb2a5727de6dd5a811ffc9ca964e9beec5efb64f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe822a_d185_46d0_b899_87c01749f371.slice/crio-d2c04dc359d7d49b9a650ddd57e9ac23af7326895b6bfb43acc4481a4469f14b\": RecentStats: unable to find data in memory cache]"
Oct 31 00:51:42.183167 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.183152    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7"} err="failed to get container status \"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7\": rpc error: code = NotFound desc = could not find container \"f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7\": container with ID starting with f66340055920d5491b1576964ea210dc4619872f79b80d438d684f00f7d6cec7 not found: ID does not exist"
Oct 31 00:51:42.183167 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.183167    4553 scope.go:122] "RemoveContainer" containerID="dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"
Oct 31 00:51:42.183378 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.183360    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684"} err="failed to get container status \"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684\": rpc error: code = NotFound desc = could not find container \"dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684\": container with ID starting with dd8c74d4d1213afbe392d777ec0104a47921cf1cb524bc7014ec38843e587684 not found: ID does not exist"
Oct 31 00:51:42.197180 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.197158    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:51:42.203466 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.203447    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9"]
Oct 31 00:51:42.320901 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.320868    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:42.461949 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.461882    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:42.669496 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.669451    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:42.809459 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.809383    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:42.865652 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:42.865620    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-b388b031-predictor-6b6fb98846-cf2v9" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:51:43.198793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:43.198759    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" path="/var/lib/kubelet/pods/44c29a56-2eef-466a-aa63-3ae8d2a77067/volumes"
Oct 31 00:51:43.199192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:43.199179    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efe822a-d185-46d0-b899-87c01749f371" path="/var/lib/kubelet/pods/4efe822a-d185-46d0-b899-87c01749f371/volumes"
Oct 31 00:51:43.242647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:43.242611    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:43.379125 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:43.379085    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:46.205821 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:46.205792    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:47.089971 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:47.089940    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:47.236253 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:47.236207    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:49.420159 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:49.420123    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:49.557240 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:49.557201    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:50.279650 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:50.279615    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:57.129434 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:51:57.129400    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:51:57.501571 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:57.501541    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:51:57.639939 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:51:57.639910    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:27.129613 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:52:27.129529    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:52:32.933032 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:32.932995    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:33.550385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:33.550345    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:36.893806 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:36.893746    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:37.113404 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:37.113372    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:39.059977 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:39.059939    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:46.253271 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:46.253243    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:47.429220 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:47.429174    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:47.599032 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:47.598993    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:47.734146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:52:47.734054    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:52:57.129881 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:52:57.129844    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:53:18.905398 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:18.905360    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:19.047583 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:19.047544    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:20.594332 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594294    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594722    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594740    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594752    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594759    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594787    4553 cpu_manager.go:414] "containerMap: removing container" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerName="kserve-container"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594794    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerName="kserve-container"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594804    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kserve-container"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594812    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kserve-container"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594888    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kube-rbac-proxy"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594899    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="agent"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594912    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c29a56-2eef-466a-aa63-3ae8d2a77067" containerName="kserve-container"
Oct 31 00:53:20.594915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594920    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe822a-d185-46d0-b899-87c01749f371" containerName="kserve-container"
Oct 31 00:53:20.595460 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.594909    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:20.597969 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.597952    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:20.604191 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.604164    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:53:20.606903 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.606884    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:53:20.607052 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.606950    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:53:20.608380 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.608356    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:53:20.683594 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.683564    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7kh\" (UniqueName: \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\") pod \"proxy-routing-2a575c77-sink-696d5fd66d-w7jh2\" (UID: \"9fad2697-c3cd-4806-9fbc-e8ac29daeadb\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:20.737180 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.737153    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:20.784177 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.784151    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7kh\" (UniqueName: \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\") pod \"proxy-routing-2a575c77-sink-696d5fd66d-w7jh2\" (UID: \"9fad2697-c3cd-4806-9fbc-e8ac29daeadb\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:20.799287 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.799258    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7kh\" (UniqueName: \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\") pod \"proxy-routing-2a575c77-sink-696d5fd66d-w7jh2\" (UID: \"9fad2697-c3cd-4806-9fbc-e8ac29daeadb\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:20.911535 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:20.911513    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:21.185312 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.185280    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:53:21.187876 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:53:21.187845    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fad2697_c3cd_4806_9fbc_e8ac29daeadb.slice/crio-c8832bfd48446045c3b07c76fe32b60d79a4a6eb4028a0fb5b110419488349bd WatchSource:0}: Error finding container c8832bfd48446045c3b07c76fe32b60d79a4a6eb4028a0fb5b110419488349bd: Status 404 returned error can't find the container with id c8832bfd48446045c3b07c76fe32b60d79a4a6eb4028a0fb5b110419488349bd
Oct 31 00:53:21.489533 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.489455    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" event={"ID":"9fad2697-c3cd-4806-9fbc-e8ac29daeadb","Type":"ContainerStarted","Data":"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"}
Oct 31 00:53:21.489533 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.489512    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" event={"ID":"9fad2697-c3cd-4806-9fbc-e8ac29daeadb","Type":"ContainerStarted","Data":"c8832bfd48446045c3b07c76fe32b60d79a4a6eb4028a0fb5b110419488349bd"}
Oct 31 00:53:21.489744 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.489652    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:21.491096 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.491069    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/\": dial tcp 10.133.0.36:8080: connect: connection refused"
Oct 31 00:53:21.513948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:21.513905    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" podStartSLOduration=1.5138926480000001 podStartE2EDuration="1.513892648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:21.513354951 +0000 UTC m=+1464.422357606" watchObservedRunningTime="2026-10-31 00:53:21.513892648 +0000 UTC m=+1464.422895302"
Oct 31 00:53:22.494112 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:22.494088    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:53:22.986986 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:22.986950    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:23.132594 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:23.132558    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:25.232802 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:25.232763    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:25.364264 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:25.364232    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:25.683216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:25.683180    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:26.006963 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.006893    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:53:26.010291 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.010274    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.016664 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.016329    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-2a575c77-kube-rbac-proxy-sar-config\""
Oct 31 00:53:26.016942 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.016924    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-2a575c77-predictor-serving-cert\""
Oct 31 00:53:26.017027 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.017013    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-2a575c77-dockercfg-fgx5h\""
Oct 31 00:53:26.017401 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.017383    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.017491 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.017412    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.017491 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.017443    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l682\" (UniqueName: \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.024248 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.024231    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:53:26.117987 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.117954    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.117987 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.117991    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.118225 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.118021    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l682\" (UniqueName: \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.118611 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.118587    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.120493 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.120471    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:26.132957 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:26.132931    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l682\" (UniqueName: \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:27.129272 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:53:27.129233    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:53:27.312192 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:27.312150    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:27.441230 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:27.441204    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:28.438915 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.438875    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.448726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.448699    4553 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.539655 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.539627    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.539809 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.539767    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.544965 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.544936    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.712530 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.712501    4553 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/14ed1d03d678c86625337ba3071984b696aa3cde0c822c88de27850420b31a27/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:28.727500 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:28.727474    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-76586bbb75-sj7jj\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:29.025914 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.025829    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:29.307657 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.307634    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:53:29.310211 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:53:29.310182    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-3afc3b0e32bad98bf1d141d4c8dcb7f9e375d5e7fb67374a3dfa88a2cfb16d8f WatchSource:0}: Error finding container 3afc3b0e32bad98bf1d141d4c8dcb7f9e375d5e7fb67374a3dfa88a2cfb16d8f: Status 404 returned error can't find the container with id 3afc3b0e32bad98bf1d141d4c8dcb7f9e375d5e7fb67374a3dfa88a2cfb16d8f
Oct 31 00:53:29.519084 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519051    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerStarted","Data":"a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb"}
Oct 31 00:53:29.519492 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519114    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerStarted","Data":"ec65224203de5cc3a043fd6198ee1b2900c3c378dca9479054f318fcd3d12a47"}
Oct 31 00:53:29.519492 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519143    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerStarted","Data":"edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0"}
Oct 31 00:53:29.519492 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519165    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerStarted","Data":"3afc3b0e32bad98bf1d141d4c8dcb7f9e375d5e7fb67374a3dfa88a2cfb16d8f"}
Oct 31 00:53:29.519492 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519189    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:29.519492 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.519213    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:29.520952 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.520925    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/\": dial tcp 10.133.0.37:8080: connect: connection refused"
Oct 31 00:53:29.546017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:29.545975    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podStartSLOduration=7.545963683 podStartE2EDuration="7.545963683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:29.545296231 +0000 UTC m=+1472.454298888" watchObservedRunningTime="2026-10-31 00:53:29.545963683 +0000 UTC m=+1472.454966338"
Oct 31 00:53:30.522774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:30.522738    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:30.524058 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:30.524040    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:30.524440 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:30.524427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:37.534408 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:37.534381    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:53:42.952520 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:42.952448    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:53:42.956040 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:42.956023    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:42.968010 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:42.967985    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:53:43.047793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.047760    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.047963 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.047804    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.047963 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.047908    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.047963 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.047939    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.048093 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.048003    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b645k\" (UniqueName: \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.052552 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.052527    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.065266 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.065244    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.149279 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.149251    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.149414 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.149296    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.149490 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.149422    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b645k\" (UniqueName: \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.149853 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.149837    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.151826 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.151806    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.163966 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.163947    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b645k\" (UniqueName: \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\") pod \"proxy-routing-2a575c77-predictor-69c48498b9-flxzs\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.272053 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.271988    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:43.550744 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.550660    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:53:43.552844 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:53:43.552818    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25786d63_3b64_4198_a71b_fd4c0ba9dbee.slice/crio-4c5fad9bb0603e19629bc4abc2c79c8a653774c893b440ece6761918e8321c2d WatchSource:0}: Error finding container 4c5fad9bb0603e19629bc4abc2c79c8a653774c893b440ece6761918e8321c2d: Status 404 returned error can't find the container with id 4c5fad9bb0603e19629bc4abc2c79c8a653774c893b440ece6761918e8321c2d
Oct 31 00:53:43.574435 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:43.574405    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerStarted","Data":"4c5fad9bb0603e19629bc4abc2c79c8a653774c893b440ece6761918e8321c2d"}
Oct 31 00:53:44.580047 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.580013    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerStarted","Data":"9e2da1cf309f2c686af425b2091236f5b4b751b850a73a30ef556e20f7d17ba4"}
Oct 31 00:53:44.580526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.580079    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerStarted","Data":"e7e7fd7511f8a0bbaa6544abce412c3771a067f5edc6c31a31b2cea7efcb0704"}
Oct 31 00:53:44.580526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.580106    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:44.580526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.580126    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerStarted","Data":"83901d056ae1037dcf1cfaba93096451be5cc2b30abf642d3484215e6f54223d"}
Oct 31 00:53:44.580526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.580148    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:44.582616 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.582596    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:44.608859 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:44.608817    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podStartSLOduration=2.6088055150000002 podStartE2EDuration="2.608805515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:44.606766663 +0000 UTC m=+1487.515769318" watchObservedRunningTime="2026-10-31 00:53:44.608805515 +0000 UTC m=+1487.517808169"
Oct 31 00:53:45.583751 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:45.583723    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:45.585327 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:45.585309    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:46.311445 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:46.311414    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:52.597757 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:52.597731    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:53:52.665056 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:52.665025    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:53:52.665379 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:52.665328    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kserve-container" containerID="cri-o://edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0" gracePeriod=30
Oct 31 00:53:52.665610 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:52.665390    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent" containerID="cri-o://a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb" gracePeriod=30
Oct 31 00:53:52.665731 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:52.665458    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" containerID="cri-o://ec65224203de5cc3a043fd6198ee1b2900c3c378dca9479054f318fcd3d12a47" gracePeriod=30
Oct 31 00:53:53.554193 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:53.554161    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:53.620354 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:53.620315    4553 generic.go:356] "Generic (PLEG): container finished" podID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerID="ec65224203de5cc3a043fd6198ee1b2900c3c378dca9479054f318fcd3d12a47" exitCode=0
Oct 31 00:53:53.620757 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:53.620381    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerDied","Data":"ec65224203de5cc3a043fd6198ee1b2900c3c378dca9479054f318fcd3d12a47"}
Oct 31 00:53:53.692347 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:53.692305    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:53:57.129746 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:53:57.129704    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:53:57.301510 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:57.301481    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:53:57.309061 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:57.309039    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:53:57.530841 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:53:57.530799    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:00.524059 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:00.524006    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:02.530090 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:02.530051    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:07.530858 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:07.530817    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:07.531341 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:07.530934    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:54:10.523191 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:10.523155    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:12.529951 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:12.529913    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:17.530086 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:17.530047    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:20.523545 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:20.523509    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:20.523948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:20.523633    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:54:22.530663 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.530623    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 31 00:54:22.729077 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:54:22.729033    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-conmon-edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-conmon-a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:54:22.729077 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:54:22.729058    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-conmon-edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4c567_5188_4d62_aa3a_ed800cb9fe9f.slice/crio-conmon-a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:54:22.735048 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.735020    4553 generic.go:356] "Generic (PLEG): container finished" podID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerID="a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb" exitCode=0
Oct 31 00:54:22.735048 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.735046    4553 generic.go:356] "Generic (PLEG): container finished" podID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerID="edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0" exitCode=137
Oct 31 00:54:22.735241 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.735105    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerDied","Data":"a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb"}
Oct 31 00:54:22.735241 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.735182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerDied","Data":"edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0"}
Oct 31 00:54:22.853220 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.853197    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:54:22.988402 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.988376    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") "
Oct 31 00:54:22.988578 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.988460    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\") pod \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") "
Oct 31 00:54:22.988578 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.988510    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\") pod \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") "
Oct 31 00:54:22.988578 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.988548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\" (UniqueName: \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\") pod \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\" (UID: \"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f\") "
Oct 31 00:54:22.988925 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.988889    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config" pod "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" (UID: "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:22.990843 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.990820    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls" pod "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" (UID: "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:54:22.991042 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.991020    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682" pod "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" (UID: "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f"). InnerVolumeSpecName "kube-api-access-5l682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:22.991257 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:22.991241    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36" pod "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" (UID: "6fc4c567-5188-4d62-aa3a-ed800cb9fe9f"). InnerVolumeSpecName "pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:54:23.089961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.089871    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:23.089961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.089915    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:23.089961 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.089930    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l682\" (UniqueName: \"kubernetes.io/projected/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f-kube-api-access-5l682\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:23.741793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.741756    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj" event={"ID":"6fc4c567-5188-4d62-aa3a-ed800cb9fe9f","Type":"ContainerDied","Data":"3afc3b0e32bad98bf1d141d4c8dcb7f9e375d5e7fb67374a3dfa88a2cfb16d8f"}
Oct 31 00:54:23.741793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.741787    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"
Oct 31 00:54:23.742324 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.741847    4553 scope.go:122] "RemoveContainer" containerID="a886a3962bc2460487253831fc5a4c9a4a5a8671d316edd53f9429af7fc5d1cb"
Oct 31 00:54:23.749688 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.749661    4553 scope.go:122] "RemoveContainer" containerID="ec65224203de5cc3a043fd6198ee1b2900c3c378dca9479054f318fcd3d12a47"
Oct 31 00:54:23.757043 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.757023    4553 scope.go:122] "RemoveContainer" containerID="edb4e6382f1cde0521c8d760c5d27c591fb615cba4bc46b7fe6143b056bef2d0"
Oct 31 00:54:23.766729 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.766707    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:54:23.774767 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:23.774747    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-76586bbb75-sj7jj"]
Oct 31 00:54:24.797733 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.797697    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.797992    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798003    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798013    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798018    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798030    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kserve-container"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798035    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kserve-container"
Oct 31 00:54:24.798110 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798108    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="agent"
Oct 31 00:54:24.798325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798116    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kube-rbac-proxy"
Oct 31 00:54:24.798325 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.798124    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" containerName="kserve-container"
Oct 31 00:54:24.802708 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.802669    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.814289 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.814265    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:54:24.904115 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.904085    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.904115 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.904116    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.904329 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.904211    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrt8r\" (UniqueName: \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.904329 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.904282    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.904411 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.904371    4553 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.908717 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.908701    4553 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:24.921692 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:24.921648    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.004658 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.004633    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.004811 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.004667    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.004811 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.004761    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrt8r\" (UniqueName: \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.005225 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.005201    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.007074 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.007052    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.019224 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.019197    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrt8r\" (UniqueName: \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\") pod \"proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.119134 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.119067    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.206058 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.206028    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc4c567-5188-4d62-aa3a-ed800cb9fe9f" path="/var/lib/kubelet/pods/6fc4c567-5188-4d62-aa3a-ed800cb9fe9f/volumes"
Oct 31 00:54:25.397558 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.397532    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:54:25.400024 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:54:25.400000    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46ee6c7_cc85_47c2_9b49_71027426e6d0.slice/crio-86c79b0dbda5dcbc24a2075251fc43a5c04ad36037f7516283ff37f0d1a2b44b WatchSource:0}: Error finding container 86c79b0dbda5dcbc24a2075251fc43a5c04ad36037f7516283ff37f0d1a2b44b: Status 404 returned error can't find the container with id 86c79b0dbda5dcbc24a2075251fc43a5c04ad36037f7516283ff37f0d1a2b44b
Oct 31 00:54:25.752947 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.752907    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerStarted","Data":"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"}
Oct 31 00:54:25.753146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.752980    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerStarted","Data":"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"}
Oct 31 00:54:25.753146 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.753016    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.753259 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.753163    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.753259 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.753210    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerStarted","Data":"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"}
Oct 31 00:54:25.753259 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.753239    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerStarted","Data":"86c79b0dbda5dcbc24a2075251fc43a5c04ad36037f7516283ff37f0d1a2b44b"}
Oct 31 00:54:25.755151 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.755130    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:25.780600 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:25.780547    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podStartSLOduration=1.780531849 podStartE2EDuration="1.780531849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:25.779756901 +0000 UTC m=+1528.688759556" watchObservedRunningTime="2026-10-31 00:54:25.780531849 +0000 UTC m=+1528.689534505"
Oct 31 00:54:26.757290 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:26.757259    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:26.758740 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:26.758720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:27.130286 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:54:27.130201    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:54:33.770252 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:33.770227    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:54:33.836793 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:33.836762    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:54:33.837076 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:33.837051    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kserve-container" containerID="cri-o://83901d056ae1037dcf1cfaba93096451be5cc2b30abf642d3484215e6f54223d" gracePeriod=30
Oct 31 00:54:33.837178 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:33.837138    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" containerID="cri-o://e7e7fd7511f8a0bbaa6544abce412c3771a067f5edc6c31a31b2cea7efcb0704" gracePeriod=30
Oct 31 00:54:33.837252 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:33.837127    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent" containerID="cri-o://9e2da1cf309f2c686af425b2091236f5b4b751b850a73a30ef556e20f7d17ba4" gracePeriod=30
Oct 31 00:54:34.791526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:34.791492    4553 generic.go:356] "Generic (PLEG): container finished" podID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerID="e7e7fd7511f8a0bbaa6544abce412c3771a067f5edc6c31a31b2cea7efcb0704" exitCode=0
Oct 31 00:54:34.791943 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:34.791564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerDied","Data":"e7e7fd7511f8a0bbaa6544abce412c3771a067f5edc6c31a31b2cea7efcb0704"}
Oct 31 00:54:35.585228 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:35.585184    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:37.594363 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:37.594316    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:54:42.594886 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:42.594844    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:54:45.585136 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:45.585095    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:46.387017 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:46.386981    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:54:47.594901 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:47.594864    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:54:47.595316 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:47.594994    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:54:48.812182 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:48.812144    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:54:52.594989 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:52.594945    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:54:55.584935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:55.584891    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:54:55.585451 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:55.585067    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:54:56.268357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:56.268326    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:54:56.415363 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:56.415329    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:54:57.128712 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:54:57.128656    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:54:57.594320 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:54:57.594281    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:55:02.594541 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:02.594500    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 31 00:55:03.906709 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:03.906656    4553 generic.go:356] "Generic (PLEG): container finished" podID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerID="9e2da1cf309f2c686af425b2091236f5b4b751b850a73a30ef556e20f7d17ba4" exitCode=0
Oct 31 00:55:03.906709 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:03.906706    4553 generic.go:356] "Generic (PLEG): container finished" podID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerID="83901d056ae1037dcf1cfaba93096451be5cc2b30abf642d3484215e6f54223d" exitCode=137
Oct 31 00:55:03.907044 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:03.906712    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerDied","Data":"9e2da1cf309f2c686af425b2091236f5b4b751b850a73a30ef556e20f7d17ba4"}
Oct 31 00:55:03.907044 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:03.906781    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerDied","Data":"83901d056ae1037dcf1cfaba93096451be5cc2b30abf642d3484215e6f54223d"}
Oct 31 00:55:04.015230 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.015208    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:55:04.032984 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.032958    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\" (UniqueName: \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\") pod \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") "
Oct 31 00:55:04.033106 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.032999    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\" (UniqueName: \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\") pod \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") "
Oct 31 00:55:04.033106 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.033097    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") "
Oct 31 00:55:04.035183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.035152    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k" pod "25786d63-3b64-4198-a71b-fd4c0ba9dbee" (UID: "25786d63-3b64-4198-a71b-fd4c0ba9dbee"). InnerVolumeSpecName "kube-api-access-b645k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:04.035283 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.035186    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls" pod "25786d63-3b64-4198-a71b-fd4c0ba9dbee" (UID: "25786d63-3b64-4198-a71b-fd4c0ba9dbee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:04.035921 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.035902    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36" pod "25786d63-3b64-4198-a71b-fd4c0ba9dbee" (UID: "25786d63-3b64-4198-a71b-fd4c0ba9dbee"). InnerVolumeSpecName "pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:55:04.133935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.133863    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\") pod \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\" (UID: \"25786d63-3b64-4198-a71b-fd4c0ba9dbee\") "
Oct 31 00:55:04.134059 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.133992    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25786d63-3b64-4198-a71b-fd4c0ba9dbee-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:04.134059 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.134005    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b645k\" (UniqueName: \"kubernetes.io/projected/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-api-access-b645k\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:04.134193 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.134172    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config" pod "25786d63-3b64-4198-a71b-fd4c0ba9dbee" (UID: "25786d63-3b64-4198-a71b-fd4c0ba9dbee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:04.235158 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.235126    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25786d63-3b64-4198-a71b-fd4c0ba9dbee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:04.914174 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.914151    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"
Oct 31 00:55:04.914174 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.914151    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs" event={"ID":"25786d63-3b64-4198-a71b-fd4c0ba9dbee","Type":"ContainerDied","Data":"4c5fad9bb0603e19629bc4abc2c79c8a653774c893b440ece6761918e8321c2d"}
Oct 31 00:55:04.914647 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.914230    4553 scope.go:122] "RemoveContainer" containerID="9e2da1cf309f2c686af425b2091236f5b4b751b850a73a30ef556e20f7d17ba4"
Oct 31 00:55:04.922796 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.922780    4553 scope.go:122] "RemoveContainer" containerID="e7e7fd7511f8a0bbaa6544abce412c3771a067f5edc6c31a31b2cea7efcb0704"
Oct 31 00:55:04.930636 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.930618    4553 scope.go:122] "RemoveContainer" containerID="83901d056ae1037dcf1cfaba93096451be5cc2b30abf642d3484215e6f54223d"
Oct 31 00:55:04.941877 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.941854    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:55:04.952527 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:04.952505    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-69c48498b9-flxzs"]
Oct 31 00:55:05.200434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.200356    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" path="/var/lib/kubelet/pods/25786d63-3b64-4198-a71b-fd4c0ba9dbee/volumes"
Oct 31 00:55:05.421955 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.421925    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:55:05.422224 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.422199    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerName="kserve-container" containerID="cri-o://d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8" gracePeriod=30
Oct 31 00:55:05.474617 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.474536    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:05.539408 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.539382    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:55:05.539701 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.539662    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kserve-container" containerID="cri-o://8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079" gracePeriod=30
Oct 31 00:55:05.539769 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.539699    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent" containerID="cri-o://304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1" gracePeriod=30
Oct 31 00:55:05.539769 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.539734    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" containerID="cri-o://b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183" gracePeriod=30
Oct 31 00:55:05.615304 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.615271    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:05.920248 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.920219    4553 generic.go:356] "Generic (PLEG): container finished" podID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerID="b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183" exitCode=0
Oct 31 00:55:05.920630 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:05.920288    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerDied","Data":"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"}
Oct 31 00:55:06.758594 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:06.758551    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:55:08.765834 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:08.765747    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:13.766370 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:13.766324    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:16.757663 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:16.757623    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:55:18.766357 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:18.766320    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:18.766743 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:18.766430    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:55:23.765985 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:23.765940    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:25.000970 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:25.000933    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:26.757885 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:26.757850    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:55:26.758375 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:26.757971    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:55:27.129141 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:27.129065    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:55:28.766851 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:28.766799    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:33.766463 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:33.766423    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Oct 31 00:55:35.629172 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.629145    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:55:35.669183 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.669159    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\" (UniqueName: \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\") pod \"9fad2697-c3cd-4806-9fbc-e8ac29daeadb\" (UID: \"9fad2697-c3cd-4806-9fbc-e8ac29daeadb\") "
Oct 31 00:55:35.671301 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.671275    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh" pod "9fad2697-c3cd-4806-9fbc-e8ac29daeadb" (UID: "9fad2697-c3cd-4806-9fbc-e8ac29daeadb"). InnerVolumeSpecName "kube-api-access-pn7kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:35.713908 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.713884    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:55:35.769835 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.769812    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\") pod \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") "
Oct 31 00:55:35.769975 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.769851    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\" (UniqueName: \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\") pod \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") "
Oct 31 00:55:35.770019 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.769985    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") pod \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") "
Oct 31 00:55:35.770082 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.770063    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\") pod \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\" (UID: \"e46ee6c7-cc85-47c2-9b49-71027426e6d0\") "
Oct 31 00:55:35.770216 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.770197    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config" pod "e46ee6c7-cc85-47c2-9b49-71027426e6d0" (UID: "e46ee6c7-cc85-47c2-9b49-71027426e6d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:35.770294 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.770278    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn7kh\" (UniqueName: \"kubernetes.io/projected/9fad2697-c3cd-4806-9fbc-e8ac29daeadb-kube-api-access-pn7kh\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:35.770348 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.770298    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:35.772171 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.772134    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r" pod "e46ee6c7-cc85-47c2-9b49-71027426e6d0" (UID: "e46ee6c7-cc85-47c2-9b49-71027426e6d0"). InnerVolumeSpecName "kube-api-access-rrt8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:35.772272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.772166    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls" pod "e46ee6c7-cc85-47c2-9b49-71027426e6d0" (UID: "e46ee6c7-cc85-47c2-9b49-71027426e6d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:35.772803 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.772783    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36" pod "e46ee6c7-cc85-47c2-9b49-71027426e6d0" (UID: "e46ee6c7-cc85-47c2-9b49-71027426e6d0"). InnerVolumeSpecName "pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 00:55:35.870552 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.870528    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e46ee6c7-cc85-47c2-9b49-71027426e6d0-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:35.870552 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.870552    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrt8r\" (UniqueName: \"kubernetes.io/projected/e46ee6c7-cc85-47c2-9b49-71027426e6d0-kube-api-access-rrt8r\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:35.870715 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.870579    4553 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" "
Oct 31 00:55:35.879687 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.879656    4553 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36") on node "ip-10-0-135-193.us-east-2.compute.internal"
Oct 31 00:55:35.970880 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:35.970823    4553 reconciler_common.go:299] "Volume detached for volume \"pvc-24de71a0-5b08-4096-b61a-fec2daabbf0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056399056a341ae36\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"csi-44ecea74dbdb1011ab7e15ca8dffe755f02760b6ccea3b3560bc6ce506a3277a\""
Oct 31 00:55:36.039365 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.039334    4553 generic.go:356] "Generic (PLEG): container finished" podID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerID="d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8" exitCode=137
Oct 31 00:55:36.039485 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.039382    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"
Oct 31 00:55:36.039485 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.039422    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" event={"ID":"9fad2697-c3cd-4806-9fbc-e8ac29daeadb","Type":"ContainerDied","Data":"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"}
Oct 31 00:55:36.039485 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.039480    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2" event={"ID":"9fad2697-c3cd-4806-9fbc-e8ac29daeadb","Type":"ContainerDied","Data":"c8832bfd48446045c3b07c76fe32b60d79a4a6eb4028a0fb5b110419488349bd"}
Oct 31 00:55:36.039629 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.039511    4553 scope.go:122] "RemoveContainer" containerID="d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"
Oct 31 00:55:36.041499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041477    4553 generic.go:356] "Generic (PLEG): container finished" podID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerID="304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1" exitCode=137
Oct 31 00:55:36.041499 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041498    4553 generic.go:356] "Generic (PLEG): container finished" podID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerID="8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079" exitCode=137
Oct 31 00:55:36.041666 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041526    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"
Oct 31 00:55:36.041666 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041525    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerDied","Data":"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"}
Oct 31 00:55:36.041666 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041579    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerDied","Data":"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"}
Oct 31 00:55:36.041666 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.041615    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf" event={"ID":"e46ee6c7-cc85-47c2-9b49-71027426e6d0","Type":"ContainerDied","Data":"86c79b0dbda5dcbc24a2075251fc43a5c04ad36037f7516283ff37f0d1a2b44b"}
Oct 31 00:55:36.048385 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.048290    4553 scope.go:122] "RemoveContainer" containerID="d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"
Oct 31 00:55:36.048612 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:36.048593    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8\": container with ID starting with d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8 not found: ID does not exist" containerID="d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"
Oct 31 00:55:36.048732 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.048623    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8"} err="failed to get container status \"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8\": rpc error: code = NotFound desc = could not find container \"d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8\": container with ID starting with d555d45cbc7b7b61fbe283767a4b559aebf78e26cde01ee291b9f663b9d04fc8 not found: ID does not exist"
Oct 31 00:55:36.048732 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.048645    4553 scope.go:122] "RemoveContainer" containerID="304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"
Oct 31 00:55:36.056447 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.056430    4553 scope.go:122] "RemoveContainer" containerID="b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"
Oct 31 00:55:36.063160 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.063146    4553 scope.go:122] "RemoveContainer" containerID="8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"
Oct 31 00:55:36.068330 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.068313    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:55:36.070167 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070156    4553 scope.go:122] "RemoveContainer" containerID="304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"
Oct 31 00:55:36.070423 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:36.070405    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1\": container with ID starting with 304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1 not found: ID does not exist" containerID="304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"
Oct 31 00:55:36.070482 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070428    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"} err="failed to get container status \"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1\": rpc error: code = NotFound desc = could not find container \"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1\": container with ID starting with 304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1 not found: ID does not exist"
Oct 31 00:55:36.070482 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070443    4553 scope.go:122] "RemoveContainer" containerID="b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"
Oct 31 00:55:36.070640 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:36.070624    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183\": container with ID starting with b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183 not found: ID does not exist" containerID="b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"
Oct 31 00:55:36.070706 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070644    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"} err="failed to get container status \"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183\": rpc error: code = NotFound desc = could not find container \"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183\": container with ID starting with b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183 not found: ID does not exist"
Oct 31 00:55:36.070706 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070656    4553 scope.go:122] "RemoveContainer" containerID="8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"
Oct 31 00:55:36.070972 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:36.070952    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079\": container with ID starting with 8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079 not found: ID does not exist" containerID="8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"
Oct 31 00:55:36.071013 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070978    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"} err="failed to get container status \"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079\": rpc error: code = NotFound desc = could not find container \"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079\": container with ID starting with 8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079 not found: ID does not exist"
Oct 31 00:55:36.071013 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.070991    4553 scope.go:122] "RemoveContainer" containerID="304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"
Oct 31 00:55:36.071185 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.071163    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1"} err="failed to get container status \"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1\": rpc error: code = NotFound desc = could not find container \"304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1\": container with ID starting with 304a76721e3a8083b367d39f5ba0de3f935d63f2d698929d119f1648495f44b1 not found: ID does not exist"
Oct 31 00:55:36.071227 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.071188    4553 scope.go:122] "RemoveContainer" containerID="b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"
Oct 31 00:55:36.071429 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.071412    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183"} err="failed to get container status \"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183\": rpc error: code = NotFound desc = could not find container \"b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183\": container with ID starting with b5326a8f7b366cfbd827c1bfddcd83eae9923d71e0e58e02bae35dea55d48183 not found: ID does not exist"
Oct 31 00:55:36.071429 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.071429    4553 scope.go:122] "RemoveContainer" containerID="8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"
Oct 31 00:55:36.071640 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.071624    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079"} err="failed to get container status \"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079\": rpc error: code = NotFound desc = could not find container \"8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079\": container with ID starting with 8c6693c9919e2e35a46e8e689ad1ff0084614ac0143dbfe2cb1b26b4087bd079 not found: ID does not exist"
Oct 31 00:55:36.074272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.074253    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-sink-696d5fd66d-w7jh2"]
Oct 31 00:55:36.091218 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.091199    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:55:36.094221 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.094202    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-2a575c77-predictor-5d655bd7f8-br8kf"]
Oct 31 00:55:36.222572 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.222479    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:36.356472 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.356441    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:36.833227 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833198    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833463    4553 cpu_manager.go:414] "containerMap: removing container" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833473    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833485    4553 cpu_manager.go:414] "containerMap: removing container" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833490    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833497    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833502    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833514    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833519    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833524    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833531    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833544    4553 cpu_manager.go:414] "containerMap: removing container" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833549    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833555    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833559    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833603    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833611    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833618    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833623    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="agent"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833629    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="kube-rbac-proxy"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833634    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" containerName="kserve-container"
Oct 31 00:55:36.833747 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.833640    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="25786d63-3b64-4198-a71b-fd4c0ba9dbee" containerName="agent"
Oct 31 00:55:36.838125 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.838106    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.845597 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.845568    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"message-dumper-predictor-serving-cert\""
Oct 31 00:55:36.845935 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.845896    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:55:36.846035 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.845955    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Oct 31 00:55:36.846220 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.846134    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:55:36.846271 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.846227    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:55:36.849967 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.849948    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:55:36.867683 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.867654    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:36.877736 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.877715    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.877862 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.877747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.877862 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.877851    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwcv\" (UniqueName: \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.978433 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.978405    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwcv\" (UniqueName: \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.978600 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.978451    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.978600 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.978473    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.979239 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.979152    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.981010 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.980984    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:36.994631 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:36.994608    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwcv\" (UniqueName: \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\") pod \"message-dumper-predictor-75c6d76d8f-66mwz\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") " pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:37.019440 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.019411    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:37.154593 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.154567    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:37.199136 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.199096    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fad2697-c3cd-4806-9fbc-e8ac29daeadb" path="/var/lib/kubelet/pods/9fad2697-c3cd-4806-9fbc-e8ac29daeadb/volumes"
Oct 31 00:55:37.199552 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.199533    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46ee6c7-cc85-47c2-9b49-71027426e6d0" path="/var/lib/kubelet/pods/e46ee6c7-cc85-47c2-9b49-71027426e6d0/volumes"
Oct 31 00:55:37.433185 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.433158    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:55:37.435871 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:55:37.435829    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0273a6_988b_42c9_8463_fb4376cc6e08.slice/crio-8e3b7924b7885b277dcfef899245509b6f90d45acd72823467d298407be95578 WatchSource:0}: Error finding container 8e3b7924b7885b277dcfef899245509b6f90d45acd72823467d298407be95578: Status 404 returned error can't find the container with id 8e3b7924b7885b277dcfef899245509b6f90d45acd72823467d298407be95578
Oct 31 00:55:37.438202 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:37.438185    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:55:38.051923 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:38.051886    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerStarted","Data":"8e3b7924b7885b277dcfef899245509b6f90d45acd72823467d298407be95578"}
Oct 31 00:55:39.058567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.058494    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerStarted","Data":"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"}
Oct 31 00:55:39.058567 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.058545    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerStarted","Data":"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"}
Oct 31 00:55:39.058977 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.058703    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:39.058977 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.058833    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:39.060496 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.060477    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:39.085272 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:39.085231    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" podStartSLOduration=1.825296075 podStartE2EDuration="3.085220929s" totalImagesPullingTime="1.259924854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:39.084192444 +0000 UTC m=+1601.993195100" watchObservedRunningTime="2026-10-31 00:55:39.085220929 +0000 UTC m=+1601.994223583"
Oct 31 00:55:43.427811 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:43.427774    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:43.596947 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:43.596910    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:46.070726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:46.070697    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:55:46.449568 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:46.449531    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:46.907108 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:46.907078    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:47.053979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:55:47.053945    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:55:57.132455 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:55:57.132418    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:56:27.129328 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:56:27.129294    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:56:46.495480 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:56:46.495392    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:57.129379 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:56:57.129341    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:57:27.128991 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:57:27.128949    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:57:42.538774 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:42.538728    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-f4cda2dafd_message-dumper-predictor-75c6d76d8f-66mwz_7c0273a6-988b-42c9-8463-fb4376cc6e08/kserve-container/0.log"
Oct 31 00:57:42.890123 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:42.890052    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:57:42.890333 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:42.890282    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kserve-container" containerID="cri-o://8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4" gracePeriod=30
Oct 31 00:57:42.890333 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:42.890319    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kube-rbac-proxy" containerID="cri-o://4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1" gracePeriod=30
Oct 31 00:57:42.926726 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:42.926669    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:43.067230 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.067195    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:43.134611 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.134591    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:57:43.274612 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.274586    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\" (UniqueName: \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\") pod \"7c0273a6-988b-42c9-8463-fb4376cc6e08\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") "
Oct 31 00:57:43.274783 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.274659    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\") pod \"7c0273a6-988b-42c9-8463-fb4376cc6e08\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") "
Oct 31 00:57:43.274783 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.274705    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\") pod \"7c0273a6-988b-42c9-8463-fb4376cc6e08\" (UID: \"7c0273a6-988b-42c9-8463-fb4376cc6e08\") "
Oct 31 00:57:43.275027 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.275003    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config" pod "7c0273a6-988b-42c9-8463-fb4376cc6e08" (UID: "7c0273a6-988b-42c9-8463-fb4376cc6e08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:43.276823 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.276799    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls" pod "7c0273a6-988b-42c9-8463-fb4376cc6e08" (UID: "7c0273a6-988b-42c9-8463-fb4376cc6e08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:57:43.276926 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.276874    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv" pod "7c0273a6-988b-42c9-8463-fb4376cc6e08" (UID: "7c0273a6-988b-42c9-8463-fb4376cc6e08"). InnerVolumeSpecName "kube-api-access-stwcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:57:43.375377 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.375351    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:43.375377 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.375374    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0273a6-988b-42c9-8463-fb4376cc6e08-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:43.375526 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.375383    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stwcv\" (UniqueName: \"kubernetes.io/projected/7c0273a6-988b-42c9-8463-fb4376cc6e08-kube-api-access-stwcv\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:43.499250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499219    4553 generic.go:356] "Generic (PLEG): container finished" podID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerID="4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1" exitCode=0
Oct 31 00:57:43.499250 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499243    4553 generic.go:356] "Generic (PLEG): container finished" podID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerID="8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4" exitCode=2
Oct 31 00:57:43.499434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499275    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"
Oct 31 00:57:43.499434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499297    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerDied","Data":"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"}
Oct 31 00:57:43.499434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499354    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerDied","Data":"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"}
Oct 31 00:57:43.499434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499402    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz" event={"ID":"7c0273a6-988b-42c9-8463-fb4376cc6e08","Type":"ContainerDied","Data":"8e3b7924b7885b277dcfef899245509b6f90d45acd72823467d298407be95578"}
Oct 31 00:57:43.499434 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.499423    4553 scope.go:122] "RemoveContainer" containerID="4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"
Oct 31 00:57:43.507562 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.507542    4553 scope.go:122] "RemoveContainer" containerID="8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"
Oct 31 00:57:43.516030 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516007    4553 scope.go:122] "RemoveContainer" containerID="4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"
Oct 31 00:57:43.516309 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:57:43.516291    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1\": container with ID starting with 4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1 not found: ID does not exist" containerID="4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"
Oct 31 00:57:43.516371 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516319    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"} err="failed to get container status \"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1\": rpc error: code = NotFound desc = could not find container \"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1\": container with ID starting with 4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1 not found: ID does not exist"
Oct 31 00:57:43.516371 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516336    4553 scope.go:122] "RemoveContainer" containerID="8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"
Oct 31 00:57:43.516659 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:57:43.516639    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4\": container with ID starting with 8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4 not found: ID does not exist" containerID="8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"
Oct 31 00:57:43.516744 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516668    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"} err="failed to get container status \"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4\": rpc error: code = NotFound desc = could not find container \"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4\": container with ID starting with 8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4 not found: ID does not exist"
Oct 31 00:57:43.516744 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516715    4553 scope.go:122] "RemoveContainer" containerID="4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"
Oct 31 00:57:43.516979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516959    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1"} err="failed to get container status \"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1\": rpc error: code = NotFound desc = could not find container \"4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1\": container with ID starting with 4e57a943965ee36086becde19813d3ec227c7f046ccde8b2e9948cc555840cb1 not found: ID does not exist"
Oct 31 00:57:43.516979 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.516978    4553 scope.go:122] "RemoveContainer" containerID="8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"
Oct 31 00:57:43.517198 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.517180    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4"} err="failed to get container status \"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4\": rpc error: code = NotFound desc = could not find container \"8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4\": container with ID starting with 8a418d992be964360be5bd6d46412a9e72b22e9aef4f1e3a502df93d71d21cf4 not found: ID does not exist"
Oct 31 00:57:43.524034 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.524014    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:57:43.534477 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:43.534426    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/message-dumper-predictor-75c6d76d8f-66mwz"]
Oct 31 00:57:45.198981 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:45.198950    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" path="/var/lib/kubelet/pods/7c0273a6-988b-42c9-8463-fb4376cc6e08/volumes"
Oct 31 00:57:46.550046 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:46.550012    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:50.968968 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:57:50.968927    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:57.129240 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:57:57.129198    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:58:00.347135 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:00.347096    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:00.488441 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:00.488408    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:14.843948 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.843875    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844160    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kserve-container"
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844171    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kserve-container"
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844192    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kube-rbac-proxy"
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844198    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kube-rbac-proxy"
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844244    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kube-rbac-proxy"
Oct 31 00:58:14.844339 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.844252    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0273a6-988b-42c9-8463-fb4376cc6e08" containerName="kserve-container"
Oct 31 00:58:14.848486 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.848470    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.851671 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.851639    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:14.855008 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.854987    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 00:58:14.855924 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.855900    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:58:14.855924 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.855915    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 00:58:14.856095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.855943    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Oct 31 00:58:14.856095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.855955    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Oct 31 00:58:14.856095 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.856027    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 00:58:14.856235 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.855915    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 00:58:14.859907 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.859890    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 00:58:14.900386 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.900362    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghjk\" (UniqueName: \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.900490 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.900396    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.900490 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.900437    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.900563 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.900486    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.900640 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.900625    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:14.995756 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:14.995731    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:15.001743 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.001721    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.001847 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.001749    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghjk\" (UniqueName: \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.001847 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.001773    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.001847 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.001800    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.001995 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.001921    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.002306 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.002284    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.002479 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.002464    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.002521 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.002468    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.004393 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.004373    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.016483 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.016456    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghjk\" (UniqueName: \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\") pod \"isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.166855 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.166830    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 00:58:15.450737 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.450707    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 00:58:15.453001 ip-10-0-135-193 kubenswrapper[4553]: W1031 00:58:15.452965    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f1ced2_3fe0_491b_ba1e_ce873662cdfa.slice/crio-0db34f04c864886c2fa7383c770aeb5271d18602af8846030bd381c6b5e97f96 WatchSource:0}: Error finding container 0db34f04c864886c2fa7383c770aeb5271d18602af8846030bd381c6b5e97f96: Status 404 returned error can't find the container with id 0db34f04c864886c2fa7383c770aeb5271d18602af8846030bd381c6b5e97f96
Oct 31 00:58:15.613027 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:15.612996    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerStarted","Data":"0db34f04c864886c2fa7383c770aeb5271d18602af8846030bd381c6b5e97f96"}
Oct 31 00:58:19.138240 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:19.138205    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:19.630996 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:19.630953    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerStarted","Data":"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"}
Oct 31 00:58:26.314225 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:26.314189    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:27.129333 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:58:27.129297    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:58:33.921823 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:33.921785    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:37.246188 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:37.246147    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:37.384591 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:37.384559    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:46.604543 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:46.604513    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:49.733603 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:49.733571    4553 generic.go:356] "Generic (PLEG): container finished" podID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerID="509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823" exitCode=0
Oct 31 00:58:49.734036 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:49.733631    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerDied","Data":"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"}
Oct 31 00:58:57.134666 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:58:57.134624    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:58:57.332842 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:57.332811    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:58:57.335268 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:58:57.335244    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 00:59:27.137888 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:59:27.137849    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 00:59:37.688024 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:59:37.687988    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:59:46.665907 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:59:46.665865    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:59:47.138386 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:59:47.138301    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:59:47.284054 ip-10-0-135-193 kubenswrapper[4553]: I1031 00:59:47.284017    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 00:59:57.133266 ip-10-0-135-193 kubenswrapper[4553]: E1031 00:59:57.133225    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:00:27.131969 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:00:27.131928    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:00:37.173247 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.173216    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerStarted","Data":"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"}
Oct 31 01:00:37.173808 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.173269    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerStarted","Data":"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"}
Oct 31 01:00:37.173808 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.173427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:00:37.173808 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.173457    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:00:37.175095 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.175073    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Oct 31 01:00:37.201982 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:37.201941    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podStartSLOduration=3.316149963 podStartE2EDuration="2m23.201930051s" totalImagesPullingTime="1m49.999164268s" totalInitContainerRuntime="29.88661582s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:37.201308071 +0000 UTC m=+1900.110310726" watchObservedRunningTime="2026-10-31 01:00:37.201930051 +0000 UTC m=+1900.110932705"
Oct 31 01:00:38.177068 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:38.177025    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Oct 31 01:00:43.181620 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:43.181594    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:00:43.182108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:43.182068    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Oct 31 01:00:46.727110 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:46.727069    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:00:53.183493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:00:53.183463    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:00:57.130384 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:00:57.130342    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:01:01.419649 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:01.419616    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 01:01:01.420191 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:01.419994    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kube-rbac-proxy" containerID="cri-o://a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30" gracePeriod=30
Oct 31 01:01:01.420902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:01.420833    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container" containerID="cri-o://a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c" gracePeriod=30
Oct 31 01:01:01.471536 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:01.471505    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:02.265639 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.265608    4553 generic.go:356] "Generic (PLEG): container finished" podID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerID="a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30" exitCode=0
Oct 31 01:01:02.265832 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.265658    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerDied","Data":"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"}
Oct 31 01:01:02.888523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.888458    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:01:02.978534 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978510    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\" (UniqueName: \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\") pod \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") "
Oct 31 01:01:02.978707 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\") pod \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") "
Oct 31 01:01:02.978707 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978581    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\") pod \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") "
Oct 31 01:01:02.978837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978720    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\") pod \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") "
Oct 31 01:01:02.978837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978765    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\") pod \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\" (UID: \"10f1ced2-3fe0-491b-ba1e-ce873662cdfa\") "
Oct 31 01:01:02.978966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978948    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config" pod "10f1ced2-3fe0-491b-ba1e-ce873662cdfa" (UID: "10f1ced2-3fe0-491b-ba1e-ce873662cdfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:02.979022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.978963    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert" pod "10f1ced2-3fe0-491b-ba1e-ce873662cdfa" (UID: "10f1ced2-3fe0-491b-ba1e-ce873662cdfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:01:02.980861 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.980830    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls" pod "10f1ced2-3fe0-491b-ba1e-ce873662cdfa" (UID: "10f1ced2-3fe0-491b-ba1e-ce873662cdfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:01:02.980983 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.980896    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk" pod "10f1ced2-3fe0-491b-ba1e-ce873662cdfa" (UID: "10f1ced2-3fe0-491b-ba1e-ce873662cdfa"). InnerVolumeSpecName "kube-api-access-bghjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:02.983060 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:02.983033    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location" pod "10f1ced2-3fe0-491b-ba1e-ce873662cdfa" (UID: "10f1ced2-3fe0-491b-ba1e-ce873662cdfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:03.080047 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.080014    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghjk\" (UniqueName: \"kubernetes.io/projected/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-api-access-bghjk\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:03.080047 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.080042    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:03.080047 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.080054    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:03.080269 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.080063    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:03.080269 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.080072    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f1ced2-3fe0-491b-ba1e-ce873662cdfa-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:03.270302 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.270271    4553 generic.go:356] "Generic (PLEG): container finished" podID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerID="a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c" exitCode=0
Oct 31 01:01:03.270433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.270322    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerDied","Data":"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"}
Oct 31 01:01:03.270433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.270332    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"
Oct 31 01:01:03.270433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.270371    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg" event={"ID":"10f1ced2-3fe0-491b-ba1e-ce873662cdfa","Type":"ContainerDied","Data":"0db34f04c864886c2fa7383c770aeb5271d18602af8846030bd381c6b5e97f96"}
Oct 31 01:01:03.270433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.270396    4553 scope.go:122] "RemoveContainer" containerID="a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"
Oct 31 01:01:03.278372 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.278356    4553 scope.go:122] "RemoveContainer" containerID="a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"
Oct 31 01:01:03.285646 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.285629    4553 scope.go:122] "RemoveContainer" containerID="509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"
Oct 31 01:01:03.292556 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.292537    4553 scope.go:122] "RemoveContainer" containerID="a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"
Oct 31 01:01:03.292888 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:01:03.292858    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30\": container with ID starting with a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30 not found: ID does not exist" containerID="a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"
Oct 31 01:01:03.293006 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.292929    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30"} err="failed to get container status \"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30\": rpc error: code = NotFound desc = could not find container \"a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30\": container with ID starting with a06ccc456d1c643839dbbe461cab9ccd210f6d11e603ffba76166d8363149e30 not found: ID does not exist"
Oct 31 01:01:03.293006 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.292950    4553 scope.go:122] "RemoveContainer" containerID="a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"
Oct 31 01:01:03.294002 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:01:03.293980    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c\": container with ID starting with a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c not found: ID does not exist" containerID="a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"
Oct 31 01:01:03.294083 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.294005    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c"} err="failed to get container status \"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c\": rpc error: code = NotFound desc = could not find container \"a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c\": container with ID starting with a2beb03cb95c7256a249d433439ab6079ecb346ceb5a57ae4c6e5a494a12f02c not found: ID does not exist"
Oct 31 01:01:03.294083 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.294020    4553 scope.go:122] "RemoveContainer" containerID="509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"
Oct 31 01:01:03.294308 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:01:03.294286    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823\": container with ID starting with 509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823 not found: ID does not exist" containerID="509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"
Oct 31 01:01:03.294350 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.294318    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823"} err="failed to get container status \"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823\": rpc error: code = NotFound desc = could not find container \"509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823\": container with ID starting with 509fee86ef796c8de78e6404ebc4f9f251f80c71e6d2bab0c9991d908c531823 not found: ID does not exist"
Oct 31 01:01:03.295132 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.295115    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 01:01:03.301093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:03.301068    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v1-predictor-57b9d4c8c4-dpqgg"]
Oct 31 01:01:04.955157 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955118    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955533    4553 cpu_manager.go:414] "containerMap: removing container" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kube-rbac-proxy"
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955550    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kube-rbac-proxy"
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955574    4553 cpu_manager.go:414] "containerMap: removing container" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="storage-initializer"
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955582    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="storage-initializer"
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955599    4553 cpu_manager.go:414] "containerMap: removing container" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container"
Oct 31 01:01:04.955629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955606    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container"
Oct 31 01:01:04.955999 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955701    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kserve-container"
Oct 31 01:01:04.955999 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.955718    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" containerName="kube-rbac-proxy"
Oct 31 01:01:04.977823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.977796    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:04.989028 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.988999    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:01:04.989152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.989131    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:04.995275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995251    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 31 01:01:04.995401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995288    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:01:04.995401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995397    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:01:04.995543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995436    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:01:04.995543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995515    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:01:04.995921 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995531    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:01:04.995921 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:04.995710    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 31 01:01:05.094507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.094478    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.094670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.094530    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcnn\" (UniqueName: \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.094670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.094618    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.094670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.094650    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.094670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.094672    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.128224 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.128196    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:05.195658 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.195633    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.195778 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.195670    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.195778 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.195720    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.195778 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.195763    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.195950 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.195799    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcnn\" (UniqueName: \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.196222 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.196195    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.196426 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.196406    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.196488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.196433    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.198693 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.198650    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.199889 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.199862    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f1ced2-3fe0-491b-ba1e-ce873662cdfa" path="/var/lib/kubelet/pods/10f1ced2-3fe0-491b-ba1e-ce873662cdfa/volumes"
Oct 31 01:01:05.211524 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.211476    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcnn\" (UniqueName: \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\") pod \"isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.341745 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.341711    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:05.643369 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:05.643302    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:01:05.646110 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:01:05.646082    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e17269_565f_4efa_8d27_373e55cf9630.slice/crio-8f78261e138f7c758e0a17aa752841ed03410811bcbe3e8d71de9d621b614b67 WatchSource:0}: Error finding container 8f78261e138f7c758e0a17aa752841ed03410811bcbe3e8d71de9d621b614b67: Status 404 returned error can't find the container with id 8f78261e138f7c758e0a17aa752841ed03410811bcbe3e8d71de9d621b614b67
Oct 31 01:01:06.283894 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:06.283846    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerStarted","Data":"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"}
Oct 31 01:01:06.284268 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:06.283918    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerStarted","Data":"8f78261e138f7c758e0a17aa752841ed03410811bcbe3e8d71de9d621b614b67"}
Oct 31 01:01:18.307253 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:18.307145    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:21.341028 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:21.340994    4553 generic.go:356] "Generic (PLEG): container finished" podID="61e17269-565f-4efa-8d27-373e55cf9630" containerID="fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03" exitCode=0
Oct 31 01:01:21.341512 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:21.341066    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerDied","Data":"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"}
Oct 31 01:01:22.347471 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:22.347430    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerStarted","Data":"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"}
Oct 31 01:01:22.347881 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:22.347548    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerStarted","Data":"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"}
Oct 31 01:01:22.347881 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:22.347778    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:22.377496 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:22.377440    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" podStartSLOduration=3.623087876 podStartE2EDuration="18.377422977s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.754335101s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:22.375311371 +0000 UTC m=+1945.284314026" watchObservedRunningTime="2026-10-31 01:01:22.377422977 +0000 UTC m=+1945.286425633"
Oct 31 01:01:23.352225 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:23.352199    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:25.756549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:25.756503    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:25.895217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:25.895173    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:27.130199 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:01:27.130163    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:01:29.361111 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:29.361080    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:01:46.784772 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:01:46.784734    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:57.129978 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:01:57.129941    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:02:27.130737 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:02:27.130668    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:02:46.848761 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:02:46.848667    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:54.227517 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:02:54.227481    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:02:56.886016 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:02:56.885971    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:57.130581 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:02:57.130530    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:03:01.486485 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.486452    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:03:01.486898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.486752    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kserve-container" containerID="cri-o://d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7" gracePeriod=30
Oct 31 01:03:01.486898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.486784    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kube-rbac-proxy" containerID="cri-o://5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec" gracePeriod=30
Oct 31 01:03:01.554461 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.554429    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:01.708600 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.708569    4553 generic.go:356] "Generic (PLEG): container finished" podID="61e17269-565f-4efa-8d27-373e55cf9630" containerID="5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec" exitCode=0
Oct 31 01:03:01.708750 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:01.708642    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerDied","Data":"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"}
Oct 31 01:03:02.937974 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.937949    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:03:02.972307 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972251    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\") pod \"61e17269-565f-4efa-8d27-373e55cf9630\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") "
Oct 31 01:03:02.972432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972326    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\") pod \"61e17269-565f-4efa-8d27-373e55cf9630\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") "
Oct 31 01:03:02.972432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972369    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\") pod \"61e17269-565f-4efa-8d27-373e55cf9630\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") "
Oct 31 01:03:02.972432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972388    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\") pod \"61e17269-565f-4efa-8d27-373e55cf9630\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") "
Oct 31 01:03:02.972432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972421    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\" (UniqueName: \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\") pod \"61e17269-565f-4efa-8d27-373e55cf9630\" (UID: \"61e17269-565f-4efa-8d27-373e55cf9630\") "
Oct 31 01:03:02.972813 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972783    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert" pod "61e17269-565f-4efa-8d27-373e55cf9630" (UID: "61e17269-565f-4efa-8d27-373e55cf9630"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:03:02.972891 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.972846    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config" pod "61e17269-565f-4efa-8d27-373e55cf9630" (UID: "61e17269-565f-4efa-8d27-373e55cf9630"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:03:02.974604 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.974577    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls" pod "61e17269-565f-4efa-8d27-373e55cf9630" (UID: "61e17269-565f-4efa-8d27-373e55cf9630"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:03:02.974712 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.974617    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn" pod "61e17269-565f-4efa-8d27-373e55cf9630" (UID: "61e17269-565f-4efa-8d27-373e55cf9630"). InnerVolumeSpecName "kube-api-access-4kcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:03:02.978862 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:02.978836    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location" pod "61e17269-565f-4efa-8d27-373e55cf9630" (UID: "61e17269-565f-4efa-8d27-373e55cf9630"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:03:03.072948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.072912    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e17269-565f-4efa-8d27-373e55cf9630-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:03:03.072948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.072943    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:03:03.072948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.072953    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e17269-565f-4efa-8d27-373e55cf9630-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:03:03.073181 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.072964    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kcnn\" (UniqueName: \"kubernetes.io/projected/61e17269-565f-4efa-8d27-373e55cf9630-kube-api-access-4kcnn\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:03:03.073181 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.072973    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e17269-565f-4efa-8d27-373e55cf9630-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:03:03.720304 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.720263    4553 generic.go:356] "Generic (PLEG): container finished" podID="61e17269-565f-4efa-8d27-373e55cf9630" containerID="d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7" exitCode=0
Oct 31 01:03:03.720509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.720316    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerDied","Data":"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"}
Oct 31 01:03:03.720509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.720329    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"
Oct 31 01:03:03.720509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.720362    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8" event={"ID":"61e17269-565f-4efa-8d27-373e55cf9630","Type":"ContainerDied","Data":"8f78261e138f7c758e0a17aa752841ed03410811bcbe3e8d71de9d621b614b67"}
Oct 31 01:03:03.720509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.720387    4553 scope.go:122] "RemoveContainer" containerID="5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"
Oct 31 01:03:03.728964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.728946    4553 scope.go:122] "RemoveContainer" containerID="d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"
Oct 31 01:03:03.736420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.736405    4553 scope.go:122] "RemoveContainer" containerID="fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"
Oct 31 01:03:03.743082 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743062    4553 scope.go:122] "RemoveContainer" containerID="5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"
Oct 31 01:03:03.743337 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:03:03.743318    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec\": container with ID starting with 5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec not found: ID does not exist" containerID="5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"
Oct 31 01:03:03.743382 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743345    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec"} err="failed to get container status \"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec\": rpc error: code = NotFound desc = could not find container \"5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec\": container with ID starting with 5314b7ec55ed99969ba2729c79102e69aabe75266314f5d25281c5d9aaf85bec not found: ID does not exist"
Oct 31 01:03:03.743382 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743361    4553 scope.go:122] "RemoveContainer" containerID="d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"
Oct 31 01:03:03.743572 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:03:03.743554    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7\": container with ID starting with d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7 not found: ID does not exist" containerID="d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"
Oct 31 01:03:03.743611 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743576    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7"} err="failed to get container status \"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7\": rpc error: code = NotFound desc = could not find container \"d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7\": container with ID starting with d34726b1f12830fdfbcfb173f0076e1d8dda98a224cc0522b5c13ed91e57b9f7 not found: ID does not exist"
Oct 31 01:03:03.743611 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743590    4553 scope.go:122] "RemoveContainer" containerID="fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"
Oct 31 01:03:03.743833 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:03:03.743809    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03\": container with ID starting with fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03 not found: ID does not exist" containerID="fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"
Oct 31 01:03:03.743880 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.743841    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03"} err="failed to get container status \"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03\": rpc error: code = NotFound desc = could not find container \"fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03\": container with ID starting with fc987684ffc42f68595e914707add4e47dd1d9a2a1faa1b9d13a2682bc8a3b03 not found: ID does not exist"
Oct 31 01:03:03.746129 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.746109    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:03:03.754943 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:03.754924    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-predictor-555d7dc4cb-vk2m8"]
Oct 31 01:03:04.249789 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.249757    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:04.301363 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301335    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:03:04.301638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301623    4553 cpu_manager.go:414] "containerMap: removing container" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="storage-initializer"
Oct 31 01:03:04.301704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301643    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="storage-initializer"
Oct 31 01:03:04.301704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301660    4553 cpu_manager.go:414] "containerMap: removing container" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kserve-container"
Oct 31 01:03:04.301704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301665    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kserve-container"
Oct 31 01:03:04.301704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301672    4553 cpu_manager.go:414] "containerMap: removing container" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kube-rbac-proxy"
Oct 31 01:03:04.301704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301692    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kube-rbac-proxy"
Oct 31 01:03:04.301851 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301761    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kserve-container"
Oct 31 01:03:04.301851 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.301768    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e17269-565f-4efa-8d27-373e55cf9630" containerName="kube-rbac-proxy"
Oct 31 01:03:04.306706 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.306670    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.313124 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.312939    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:03:04.313124 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313018    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Oct 31 01:03:04.313124 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313019    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:03:04.313302 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313283    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:03:04.313505 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313486    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:03:04.313584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313527    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:03:04.313584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.313567    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-tensorflow-predictor-serving-cert\""
Oct 31 01:03:04.317475 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.317454    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:03:04.382737 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.382706    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.382737 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.382733    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.382955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.382863    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pls4m\" (UniqueName: \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.382955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.382899    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.383026 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.382966    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.391694 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.391645    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:04.484093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484064    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484207 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484113    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484207 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484130    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484207 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484187    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pls4m\" (UniqueName: \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484212    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484597 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484578    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484885    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.484966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.484885    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.486724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.486707    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.498913 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.498884    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pls4m\" (UniqueName: \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\") pod \"isvc-tensorflow-predictor-75dfc669c5-l7564\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.624152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.624060    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:04.901931 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:04.901904    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:03:04.904197 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:03:04.904171    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf58bcb_e9f3_4384_8a11_aa2a2c2c032d.slice/crio-14114b6e50b1d85c5439ab5fe06c9c07719de1362760c98a227270edde98a995 WatchSource:0}: Error finding container 14114b6e50b1d85c5439ab5fe06c9c07719de1362760c98a227270edde98a995: Status 404 returned error can't find the container with id 14114b6e50b1d85c5439ab5fe06c9c07719de1362760c98a227270edde98a995
Oct 31 01:03:05.200636 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:05.200563    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e17269-565f-4efa-8d27-373e55cf9630" path="/var/lib/kubelet/pods/61e17269-565f-4efa-8d27-373e55cf9630/volumes"
Oct 31 01:03:05.732474 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:05.732430    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerStarted","Data":"628604ea593be7b9cf3c89db90893928a005a4f45302773075015a3e8a1433b7"}
Oct 31 01:03:05.733116 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:05.733071    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerStarted","Data":"14114b6e50b1d85c5439ab5fe06c9c07719de1362760c98a227270edde98a995"}
Oct 31 01:03:06.068134 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:06.068050    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:06.282534 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:06.282499    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:10.752040 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:10.752008    4553 generic.go:356] "Generic (PLEG): container finished" podID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerID="628604ea593be7b9cf3c89db90893928a005a4f45302773075015a3e8a1433b7" exitCode=0
Oct 31 01:03:10.752451 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:10.752084    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerDied","Data":"628604ea593be7b9cf3c89db90893928a005a4f45302773075015a3e8a1433b7"}
Oct 31 01:03:10.753463 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:10.753449    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:03:14.773599 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.773521    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerStarted","Data":"fef675a4f98783b05cb7b34a2275fae3eb52dd1c8e9f7e6f3d0c15728d2c27a4"}
Oct 31 01:03:14.773599 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.773587    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerStarted","Data":"a584d4bed68a29688f8cd8ee8bfb7ffa4ce21096c7aacbf638953e99774f4a27"}
Oct 31 01:03:14.774066 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.773876    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:14.774066 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.773934    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:14.775272 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.775247    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 31 01:03:14.803287 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:14.803249    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podStartSLOduration=1.9247080890000001 podStartE2EDuration="10.803238191s" totalImagesPullingTime="3.744719894s" totalInitContainerRuntime="5.133810208s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:14.802752701 +0000 UTC m=+2057.711755354" watchObservedRunningTime="2026-10-31 01:03:14.803238191 +0000 UTC m=+2057.712240847"
Oct 31 01:03:15.777134 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:15.777093    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 31 01:03:20.782157 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:20.782128    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:20.782807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:20.782780    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 31 01:03:27.129697 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:03:27.129647    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:03:30.783380 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:30.783354    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:44.557693 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.557655    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:03:44.558137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.557900    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container" containerID="cri-o://a584d4bed68a29688f8cd8ee8bfb7ffa4ce21096c7aacbf638953e99774f4a27" gracePeriod=30
Oct 31 01:03:44.558137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.557956    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" containerID="cri-o://fef675a4f98783b05cb7b34a2275fae3eb52dd1c8e9f7e6f3d0c15728d2c27a4" gracePeriod=30
Oct 31 01:03:44.642338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.642306    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:44.892463 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.892391    4553 generic.go:356] "Generic (PLEG): container finished" podID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerID="fef675a4f98783b05cb7b34a2275fae3eb52dd1c8e9f7e6f3d0c15728d2c27a4" exitCode=0
Oct 31 01:03:44.892587 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:44.892463    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerDied","Data":"fef675a4f98783b05cb7b34a2275fae3eb52dd1c8e9f7e6f3d0c15728d2c27a4"}
Oct 31 01:03:45.777949 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:45.777909    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:03:46.920933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:46.920900    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:49.033380 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:49.033335    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:49.178391 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:49.178353    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:50.777384 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:50.777338    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:03:55.777924 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:55.777883    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:03:55.778418 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:55.777995    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:03:57.129349 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:03:57.129307    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:03:57.355375 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:57.355345    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:03:57.359360 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:03:57.359340    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:04:00.777814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:00.777762    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:04:05.777640 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:05.777599    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:04:10.778185 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:10.778102    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Oct 31 01:04:15.004734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.004670    4553 generic.go:356] "Generic (PLEG): container finished" podID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerID="a584d4bed68a29688f8cd8ee8bfb7ffa4ce21096c7aacbf638953e99774f4a27" exitCode=137
Oct 31 01:04:15.004734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.004716    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerDied","Data":"a584d4bed68a29688f8cd8ee8bfb7ffa4ce21096c7aacbf638953e99774f4a27"}
Oct 31 01:04:15.195043 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.195014    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:04:15.300120 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300045    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\") pod \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") "
Oct 31 01:04:15.300120 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300111    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\") pod \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") "
Oct 31 01:04:15.300356 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300140    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\") pod \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") "
Oct 31 01:04:15.300356 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300160    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\" (UniqueName: \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\") pod \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") "
Oct 31 01:04:15.300356 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300200    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\") pod \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\" (UID: \"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d\") "
Oct 31 01:04:15.300522 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300421    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert" pod "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" (UID: "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:15.300577 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.300540    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config" pod "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" (UID: "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:15.302588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.302564    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls" pod "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" (UID: "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:15.302588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.302568    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m" pod "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" (UID: "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d"). InnerVolumeSpecName "kube-api-access-pls4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:15.310154 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.310126    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location" pod "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" (UID: "1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:15.401128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.401087    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:15.401128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.401126    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:15.401288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.401142    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:15.401288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.401155    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pls4m\" (UniqueName: \"kubernetes.io/projected/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kube-api-access-pls4m\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:15.401288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:15.401167    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:16.009986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.009947    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564" event={"ID":"1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d","Type":"ContainerDied","Data":"14114b6e50b1d85c5439ab5fe06c9c07719de1362760c98a227270edde98a995"}
Oct 31 01:04:16.009986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.009974    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"
Oct 31 01:04:16.010548 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.010025    4553 scope.go:122] "RemoveContainer" containerID="fef675a4f98783b05cb7b34a2275fae3eb52dd1c8e9f7e6f3d0c15728d2c27a4"
Oct 31 01:04:16.018170 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.018153    4553 scope.go:122] "RemoveContainer" containerID="a584d4bed68a29688f8cd8ee8bfb7ffa4ce21096c7aacbf638953e99774f4a27"
Oct 31 01:04:16.025389 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.025370    4553 scope.go:122] "RemoveContainer" containerID="628604ea593be7b9cf3c89db90893928a005a4f45302773075015a3e8a1433b7"
Oct 31 01:04:16.036496 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.036475    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:04:16.044770 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.044750    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-predictor-75dfc669c5-l7564"]
Oct 31 01:04:16.495189 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495155    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:04:16.495502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495489    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="storage-initializer"
Oct 31 01:04:16.495551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495504    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="storage-initializer"
Oct 31 01:04:16.495551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495522    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy"
Oct 31 01:04:16.495551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495529    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy"
Oct 31 01:04:16.495551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495542    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container"
Oct 31 01:04:16.495551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495548    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container"
Oct 31 01:04:16.495749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495607    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kube-rbac-proxy"
Oct 31 01:04:16.495749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.495616    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" containerName="kserve-container"
Oct 31 01:04:16.500055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.500039    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.505700 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.505651    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:16.506270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.506200    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:04:16.506270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.506220    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:04:16.506270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.506238    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:04:16.506270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.506237    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Oct 31 01:04:16.506440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.506280    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:04:16.507704 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507652    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.507810 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507745    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Oct 31 01:04:16.507810 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507740    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.507810 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507807    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:04:16.507928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507846    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfxq\" (UniqueName: \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.507928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507874    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.507928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.507892    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.512377 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.512358    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:04:16.608283 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608250    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608306    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608339    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfxq\" (UniqueName: \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608366    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608391    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608801    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608997 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608980    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.608997 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.608987    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.610952 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.610932    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.623320 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.623301    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfxq\" (UniqueName: \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\") pod \"isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") " pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:16.648287 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.648261    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:16.818956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:16.818894    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:17.104594 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:17.104554    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:04:17.106758 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:04:17.106727    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a324729_69d9_48af_9f6d_20fa73a6958a.slice/crio-a68774ff3cf2873006b9cc89931acb94ea782dff33ad9b8c2a2e6f7fcf05c6f4 WatchSource:0}: Error finding container a68774ff3cf2873006b9cc89931acb94ea782dff33ad9b8c2a2e6f7fcf05c6f4: Status 404 returned error can't find the container with id a68774ff3cf2873006b9cc89931acb94ea782dff33ad9b8c2a2e6f7fcf05c6f4
Oct 31 01:04:17.199716 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:17.199667    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d" path="/var/lib/kubelet/pods/1cf58bcb-e9f3-4384-8a11-aa2a2c2c032d/volumes"
Oct 31 01:04:18.019789 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:18.019749    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerStarted","Data":"d85d1df6998fd9e51031a1e43067715303ec0c86dbb410491fd9f8524c157b81"}
Oct 31 01:04:18.019953 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:18.019817    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerStarted","Data":"a68774ff3cf2873006b9cc89931acb94ea782dff33ad9b8c2a2e6f7fcf05c6f4"}
Oct 31 01:04:23.040789 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:23.040755    4553 generic.go:356] "Generic (PLEG): container finished" podID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerID="d85d1df6998fd9e51031a1e43067715303ec0c86dbb410491fd9f8524c157b81" exitCode=0
Oct 31 01:04:23.041208 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:23.040796    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerDied","Data":"d85d1df6998fd9e51031a1e43067715303ec0c86dbb410491fd9f8524c157b81"}
Oct 31 01:04:24.047522 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.047488    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerStarted","Data":"f4d5060055e831d0ec9fd1e5b3d1befde05b51e10ada05df02181572db8e16d8"}
Oct 31 01:04:24.047943 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.047547    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerStarted","Data":"06124fa9e997334dc430da45a253db30cc97f3d3553765df082d56254fb12b0c"}
Oct 31 01:04:24.047943 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.047825    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:24.048015 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.047975    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:24.049524 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.049497    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 31 01:04:24.078547 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:24.078497    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podStartSLOduration=2.617828163 podStartE2EDuration="8.07848657s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.460658407s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:24.076462198 +0000 UTC m=+2126.985464851" watchObservedRunningTime="2026-10-31 01:04:24.07848657 +0000 UTC m=+2126.987489250"
Oct 31 01:04:25.052814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:25.052774    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 31 01:04:27.130063 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:04:27.130024    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:04:30.057575 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:30.057545    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:30.058050 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:30.058026    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 31 01:04:39.354869 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:39.354835    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.058748 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:40.058720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:04:47.007110 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:47.007080    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:47.710380 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:47.710342    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:47.854791 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:47.854753    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.100537 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:53.100496    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.239020 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:53.238982    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:56.760052 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:56.760023    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:04:56.760485 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:56.760290    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" containerID="cri-o://f4d5060055e831d0ec9fd1e5b3d1befde05b51e10ada05df02181572db8e16d8" gracePeriod=30
Oct 31 01:04:56.760485 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:56.760290    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container" containerID="cri-o://06124fa9e997334dc430da45a253db30cc97f3d3553765df082d56254fb12b0c" gracePeriod=30
Oct 31 01:04:56.833640 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:56.833604    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:57.128880 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:04:57.128804    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:04:57.174370 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:57.174332    4553 generic.go:356] "Generic (PLEG): container finished" podID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerID="f4d5060055e831d0ec9fd1e5b3d1befde05b51e10ada05df02181572db8e16d8" exitCode=0
Oct 31 01:04:57.174550 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:04:57.174433    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerDied","Data":"f4d5060055e831d0ec9fd1e5b3d1befde05b51e10ada05df02181572db8e16d8"}
Oct 31 01:05:00.053326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:00.053280    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:05.053372 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:05.053332    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:10.053069 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:10.053029    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:10.053570 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:10.053146    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:05:15.053900 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:15.053855    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:20.053389 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:20.053346    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:25.053719 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:25.053652    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 31 01:05:27.128867 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:05:27.128771    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:05:27.283456 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.283427    4553 generic.go:356] "Generic (PLEG): container finished" podID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerID="06124fa9e997334dc430da45a253db30cc97f3d3553765df082d56254fb12b0c" exitCode=137
Oct 31 01:05:27.283574 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.283476    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerDied","Data":"06124fa9e997334dc430da45a253db30cc97f3d3553765df082d56254fb12b0c"}
Oct 31 01:05:27.408365 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.408339    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:05:27.547245 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547210    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\") pod \"9a324729-69d9-48af-9f6d-20fa73a6958a\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") "
Oct 31 01:05:27.547439 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547254    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\") pod \"9a324729-69d9-48af-9f6d-20fa73a6958a\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") "
Oct 31 01:05:27.547439 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547317    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\") pod \"9a324729-69d9-48af-9f6d-20fa73a6958a\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") "
Oct 31 01:05:27.547439 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547353    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\") pod \"9a324729-69d9-48af-9f6d-20fa73a6958a\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") "
Oct 31 01:05:27.547439 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547370    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\" (UniqueName: \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\") pod \"9a324729-69d9-48af-9f6d-20fa73a6958a\" (UID: \"9a324729-69d9-48af-9f6d-20fa73a6958a\") "
Oct 31 01:05:27.547779 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547745    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert" pod "9a324729-69d9-48af-9f6d-20fa73a6958a" (UID: "9a324729-69d9-48af-9f6d-20fa73a6958a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:27.547857 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.547748    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config" pod "9a324729-69d9-48af-9f6d-20fa73a6958a" (UID: "9a324729-69d9-48af-9f6d-20fa73a6958a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:27.549700 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.549658    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq" pod "9a324729-69d9-48af-9f6d-20fa73a6958a" (UID: "9a324729-69d9-48af-9f6d-20fa73a6958a"). InnerVolumeSpecName "kube-api-access-rzfxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:27.549700 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.549666    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls" pod "9a324729-69d9-48af-9f6d-20fa73a6958a" (UID: "9a324729-69d9-48af-9f6d-20fa73a6958a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:27.557586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.557559    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location" pod "9a324729-69d9-48af-9f6d-20fa73a6958a" (UID: "9a324729-69d9-48af-9f6d-20fa73a6958a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:27.648543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.648481    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:27.648543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.648506    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfxq\" (UniqueName: \"kubernetes.io/projected/9a324729-69d9-48af-9f6d-20fa73a6958a-kube-api-access-rzfxq\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:27.648543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.648517    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a324729-69d9-48af-9f6d-20fa73a6958a-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:27.648543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.648525    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a324729-69d9-48af-9f6d-20fa73a6958a-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:27.648543 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:27.648534    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a324729-69d9-48af-9f6d-20fa73a6958a-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:28.288470 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.288439    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x" event={"ID":"9a324729-69d9-48af-9f6d-20fa73a6958a","Type":"ContainerDied","Data":"a68774ff3cf2873006b9cc89931acb94ea782dff33ad9b8c2a2e6f7fcf05c6f4"}
Oct 31 01:05:28.288917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.288508    4553 scope.go:122] "RemoveContainer" containerID="f4d5060055e831d0ec9fd1e5b3d1befde05b51e10ada05df02181572db8e16d8"
Oct 31 01:05:28.288917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.288445    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"
Oct 31 01:05:28.297237 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.297220    4553 scope.go:122] "RemoveContainer" containerID="06124fa9e997334dc430da45a253db30cc97f3d3553765df082d56254fb12b0c"
Oct 31 01:05:28.304276 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.304258    4553 scope.go:122] "RemoveContainer" containerID="d85d1df6998fd9e51031a1e43067715303ec0c86dbb410491fd9f8524c157b81"
Oct 31 01:05:28.315493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.315471    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:05:28.321799 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:28.321779    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-tensorflow-runtime-predictor-57499d76ff-tsw7x"]
Oct 31 01:05:29.198817 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:29.198783    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" path="/var/lib/kubelet/pods/9a324729-69d9-48af-9f6d-20fa73a6958a/volumes"
Oct 31 01:05:30.719360 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719326    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719639    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719654    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719704    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719713    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719724    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="storage-initializer"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719732    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="storage-initializer"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719836    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kube-rbac-proxy"
Oct 31 01:05:30.719897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.719857    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a324729-69d9-48af-9f6d-20fa73a6958a" containerName="kserve-container"
Oct 31 01:05:30.724791 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.724773    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.731092 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.731049    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:30.732220 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.732200    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:05:30.737295 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.737275    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:05:30.738048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.738029    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-triton-predictor-serving-cert\""
Oct 31 01:05:30.738048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.738053    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:05:30.738201 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.738060    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:05:30.832484 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.832461    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:05:30.832657 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.832645    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Oct 31 01:05:30.832732 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.832662    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:05:30.871043 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.871021    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.871149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.871056    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.871149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.871074    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.871246 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.871143    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.871283 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.871250    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p4h\" (UniqueName: \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.881048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.881024    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:30.972296 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p4h\" (UniqueName: \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.972418 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972319    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.972418 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972354    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.972418 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972378    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.972418 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972403    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.972871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.972852    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.973042 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.973023    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.973101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.973083    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.974952 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.974937    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:30.987261 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:30.987238    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p4h\" (UniqueName: \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\") pod \"isvc-triton-predictor-5f788878c6-fr8sl\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") " pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:31.044997 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:31.044975    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:05:31.322188 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:31.322164    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:05:31.324406 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:05:31.324372    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff34097f_8d9b_416d_84b3_6c89484fd061.slice/crio-a913a9a8b9ff320267c70c2ad1aecf39563a3e6c21915f35b5260a72293401b7 WatchSource:0}: Error finding container a913a9a8b9ff320267c70c2ad1aecf39563a3e6c21915f35b5260a72293401b7: Status 404 returned error can't find the container with id a913a9a8b9ff320267c70c2ad1aecf39563a3e6c21915f35b5260a72293401b7
Oct 31 01:05:32.305303 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:32.305262    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerStarted","Data":"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"}
Oct 31 01:05:32.305737 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:32.305323    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerStarted","Data":"a913a9a8b9ff320267c70c2ad1aecf39563a3e6c21915f35b5260a72293401b7"}
Oct 31 01:05:36.322742 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:36.322705    4553 generic.go:356] "Generic (PLEG): container finished" podID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerID="6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f" exitCode=0
Oct 31 01:05:36.323109 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:36.322766    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerDied","Data":"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"}
Oct 31 01:05:47.036109 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:05:47.035896    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:57.141571 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:05:57.141515    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:06:14.994859 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:14.994793    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:27.134826 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:06:27.134785    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:06:47.036982 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:47.036946    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:53.643417 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:53.643379    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:56.080720 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.080669    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:06:56.084911 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.084880    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:56.155602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.155565    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:06:56.155802 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.155690    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.161776 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.161735    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 31 01:06:56.161917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.161802    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:06:56.161917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.161839    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:06:56.161917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.161859    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:06:56.162273 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.162253    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 31 01:06:56.162385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.162354    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:06:56.162385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.162357    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:06:56.215323 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.215296    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.215480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.215355    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qbj\" (UniqueName: \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.215542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.215505    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.215604 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.215560    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.215661 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.215603    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.316909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.316864    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.316909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.316912    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qbj\" (UniqueName: \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.316939    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.316972    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.317014    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317380 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:06:56.317354    4553 secret.go:189] Couldn't get secret e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Oct 31 01:06:56.317495 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:06:56.317453    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls podName:ebf89c8c-c5fb-4f04-9654-d300a792b9ed nodeName:}" failed. No retries permitted until 2026-10-31 01:06:56.817438065 +0000 UTC m=+2279.726440698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Oct 31 01:06:56.317624 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.317606    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.317642    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.317884 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.317866    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.332210 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.332137    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qbj\" (UniqueName: \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.485457 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.485383    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:56.820896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.820861    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:56.823586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:56.823555    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:57.113896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:57.113821    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:06:57.201719 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:06:57.201660    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:06:57.556834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:57.556222    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:06:57.558451 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:06:57.558420    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf89c8c_c5fb_4f04_9654_d300a792b9ed.slice/crio-7db841bf4fd8ad28daceb9ce18d8db12d4b4b35396cd7951f020577d79da0574 WatchSource:0}: Error finding container 7db841bf4fd8ad28daceb9ce18d8db12d4b4b35396cd7951f020577d79da0574: Status 404 returned error can't find the container with id 7db841bf4fd8ad28daceb9ce18d8db12d4b4b35396cd7951f020577d79da0574
Oct 31 01:06:57.564239 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:57.564198    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:06:57.715300 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:57.715262    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerStarted","Data":"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"}
Oct 31 01:06:57.715427 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:06:57.715342    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerStarted","Data":"7db841bf4fd8ad28daceb9ce18d8db12d4b4b35396cd7951f020577d79da0574"}
Oct 31 01:07:25.842859 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:25.842818    4553 generic.go:356] "Generic (PLEG): container finished" podID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerID="abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c" exitCode=0
Oct 31 01:07:25.843399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:25.842863    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerDied","Data":"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"}
Oct 31 01:07:26.849644 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:26.849599    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerStarted","Data":"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"}
Oct 31 01:07:26.850148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:26.849735    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerStarted","Data":"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"}
Oct 31 01:07:26.850148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:26.849963    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:07:26.878448 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:26.878393    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" podStartSLOduration=3.192520618 podStartE2EDuration="30.878382313s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.685861695s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:26.878053716 +0000 UTC m=+2309.787056367" watchObservedRunningTime="2026-10-31 01:07:26.878382313 +0000 UTC m=+2309.787384966"
Oct 31 01:07:27.137476 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:07:27.137387    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:07:27.854755 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:27.854720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:07:32.881953 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.881918    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerStarted","Data":"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"}
Oct 31 01:07:32.882500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.881976    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerStarted","Data":"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"}
Oct 31 01:07:32.882500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.882241    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:32.882500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.882386    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:32.883883 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.883860    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Oct 31 01:07:32.910067 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:32.910025    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podStartSLOduration=3.337712936 podStartE2EDuration="2m2.910014125s" totalImagesPullingTime="1m55.580779324s" totalInitContainerRuntime="3.991521865s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:32.909031178 +0000 UTC m=+2315.818033832" watchObservedRunningTime="2026-10-31 01:07:32.910014125 +0000 UTC m=+2315.819016837"
Oct 31 01:07:33.865877 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:33.865842    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:07:33.887076 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:33.887045    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Oct 31 01:07:34.890737 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:34.890692    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Oct 31 01:07:39.895174 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:39.895146    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:39.896022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:39.896002    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:43.261142 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.261110    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:07:43.261570 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.261338    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container" containerID="cri-o://735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3" gracePeriod=30
Oct 31 01:07:43.261570 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.261411    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kube-rbac-proxy" containerID="cri-o://b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4" gracePeriod=30
Oct 31 01:07:43.339419 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.339383    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:43.935325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.935296    4553 generic.go:356] "Generic (PLEG): container finished" podID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerID="b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4" exitCode=0
Oct 31 01:07:43.935500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:43.935370    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerDied","Data":"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"}
Oct 31 01:07:44.890943 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:44.890904    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Oct 31 01:07:45.617033 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.617010    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:45.740654 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.740568    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\" (UniqueName: \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\") pod \"ff34097f-8d9b-416d-84b3-6c89484fd061\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") "
Oct 31 01:07:45.740654 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.740639    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\") pod \"ff34097f-8d9b-416d-84b3-6c89484fd061\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") "
Oct 31 01:07:45.741008 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.740702    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\") pod \"ff34097f-8d9b-416d-84b3-6c89484fd061\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") "
Oct 31 01:07:45.741008 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.740727    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\") pod \"ff34097f-8d9b-416d-84b3-6c89484fd061\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") "
Oct 31 01:07:45.741008 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.740748    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\") pod \"ff34097f-8d9b-416d-84b3-6c89484fd061\" (UID: \"ff34097f-8d9b-416d-84b3-6c89484fd061\") "
Oct 31 01:07:45.741177 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.741074    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config" pod "ff34097f-8d9b-416d-84b3-6c89484fd061" (UID: "ff34097f-8d9b-416d-84b3-6c89484fd061"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:45.741177 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.741109    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert" pod "ff34097f-8d9b-416d-84b3-6c89484fd061" (UID: "ff34097f-8d9b-416d-84b3-6c89484fd061"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:45.741271 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.741203    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location" pod "ff34097f-8d9b-416d-84b3-6c89484fd061" (UID: "ff34097f-8d9b-416d-84b3-6c89484fd061"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:45.755075 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.755046    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls" pod "ff34097f-8d9b-416d-84b3-6c89484fd061" (UID: "ff34097f-8d9b-416d-84b3-6c89484fd061"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:45.755212 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.755198    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h" pod "ff34097f-8d9b-416d-84b3-6c89484fd061" (UID: "ff34097f-8d9b-416d-84b3-6c89484fd061"). InnerVolumeSpecName "kube-api-access-d8p4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:45.841707 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.841662    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p4h\" (UniqueName: \"kubernetes.io/projected/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-api-access-d8p4h\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:07:45.841707 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.841705    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:07:45.841878 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.841717    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff34097f-8d9b-416d-84b3-6c89484fd061-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:07:45.841878 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.841727    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff34097f-8d9b-416d-84b3-6c89484fd061-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:07:45.841878 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.841736    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff34097f-8d9b-416d-84b3-6c89484fd061-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:07:45.944258 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.944225    4553 generic.go:356] "Generic (PLEG): container finished" podID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerID="735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3" exitCode=0
Oct 31 01:07:45.944718 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.944291    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"
Oct 31 01:07:45.944718 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.944307    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerDied","Data":"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"}
Oct 31 01:07:45.944718 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.944369    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl" event={"ID":"ff34097f-8d9b-416d-84b3-6c89484fd061","Type":"ContainerDied","Data":"a913a9a8b9ff320267c70c2ad1aecf39563a3e6c21915f35b5260a72293401b7"}
Oct 31 01:07:45.944718 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.944396    4553 scope.go:122] "RemoveContainer" containerID="b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"
Oct 31 01:07:45.958268 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.958245    4553 scope.go:122] "RemoveContainer" containerID="735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"
Oct 31 01:07:45.966051 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.966032    4553 scope.go:122] "RemoveContainer" containerID="6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"
Oct 31 01:07:45.973703 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.973664    4553 scope.go:122] "RemoveContainer" containerID="b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"
Oct 31 01:07:45.973987 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:07:45.973971    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4\": container with ID starting with b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4 not found: ID does not exist" containerID="b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"
Oct 31 01:07:45.974055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.973995    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4"} err="failed to get container status \"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4\": rpc error: code = NotFound desc = could not find container \"b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4\": container with ID starting with b0b64e51820dd1f58ad8632d58d36cdabe48bd7d1a36e38e8dde3ecace7c15a4 not found: ID does not exist"
Oct 31 01:07:45.974055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.974009    4553 scope.go:122] "RemoveContainer" containerID="735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"
Oct 31 01:07:45.974235 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:07:45.974221    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3\": container with ID starting with 735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3 not found: ID does not exist" containerID="735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"
Oct 31 01:07:45.974279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.974239    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3"} err="failed to get container status \"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3\": rpc error: code = NotFound desc = could not find container \"735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3\": container with ID starting with 735cd5037fde40dc83f44b4a04b885a10a861278fa678720287ee347b1e304d3 not found: ID does not exist"
Oct 31 01:07:45.974279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.974253    4553 scope.go:122] "RemoveContainer" containerID="6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"
Oct 31 01:07:45.974497 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:07:45.974478    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f\": container with ID starting with 6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f not found: ID does not exist" containerID="6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"
Oct 31 01:07:45.974551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.974505    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f"} err="failed to get container status \"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f\": rpc error: code = NotFound desc = could not find container \"6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f\": container with ID starting with 6838c88fa2414fe3f7e04d6924295e155136867e9e4b85314d62120bb2976f7f not found: ID does not exist"
Oct 31 01:07:45.976845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.976822    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:07:45.983371 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:45.983351    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-triton-predictor-5f788878c6-fr8sl"]
Oct 31 01:07:46.684968 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:46.684933    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:46.828630 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:46.828601    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:47.135111 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:47.135029    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:47.200508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:47.200480    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" path="/var/lib/kubelet/pods/ff34097f-8d9b-416d-84b3-6c89484fd061/volumes"
Oct 31 01:07:47.269236 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:07:47.269208    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:57.129902 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:07:57.129866    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:08:27.130394 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:08:27.130361    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:08:47.193192 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:08:47.193161    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:57.130934 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:08:57.130896    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:08:57.377311 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:08:57.377281    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:08:57.382218 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:08:57.382161    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:09:04.229782 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:04.229753    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:09:14.180203 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:14.180174    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:09:14.180641 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:14.180406    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kserve-container" containerID="cri-o://695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070" gracePeriod=30
Oct 31 01:09:14.180641 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:14.180456    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kube-rbac-proxy" containerID="cri-o://2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304" gracePeriod=30
Oct 31 01:09:14.195298 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:14.195267    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.133.0.46:8080: connect: connection refused"
Oct 31 01:09:14.282196 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:14.282165    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:15.263988 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.263957    4553 generic.go:356] "Generic (PLEG): container finished" podID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerID="2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304" exitCode=0
Oct 31 01:09:15.264410 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.264041    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerDied","Data":"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"}
Oct 31 01:09:15.622152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.622129    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:09:15.728298 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728252    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\") pod \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") "
Oct 31 01:09:15.728502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728350    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") pod \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") "
Oct 31 01:09:15.728502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728411    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\") pod \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") "
Oct 31 01:09:15.728502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728440    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\" (UniqueName: \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\") pod \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") "
Oct 31 01:09:15.728502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728459    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\") pod \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\" (UID: \"ebf89c8c-c5fb-4f04-9654-d300a792b9ed\") "
Oct 31 01:09:15.728771 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728744    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config" pod "ebf89c8c-c5fb-4f04-9654-d300a792b9ed" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:15.728854 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.728827    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert" pod "ebf89c8c-c5fb-4f04-9654-d300a792b9ed" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:15.730793 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.730768    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls" pod "ebf89c8c-c5fb-4f04-9654-d300a792b9ed" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:09:15.730902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.730876    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj" pod "ebf89c8c-c5fb-4f04-9654-d300a792b9ed" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed"). InnerVolumeSpecName "kube-api-access-l7qbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:15.732358 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.732336    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location" pod "ebf89c8c-c5fb-4f04-9654-d300a792b9ed" (UID: "ebf89c8c-c5fb-4f04-9654-d300a792b9ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:15.829898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.829809    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:09:15.829898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.829842    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:09:15.829898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.829858    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qbj\" (UniqueName: \"kubernetes.io/projected/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-api-access-l7qbj\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:09:15.829898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.829872    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:09:15.829898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:15.829883    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebf89c8c-c5fb-4f04-9654-d300a792b9ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:09:16.269123 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.269092    4553 generic.go:356] "Generic (PLEG): container finished" podID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerID="695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070" exitCode=0
Oct 31 01:09:16.269609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.269145    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerDied","Data":"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"}
Oct 31 01:09:16.269609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.269152    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"
Oct 31 01:09:16.269609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.269193    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2" event={"ID":"ebf89c8c-c5fb-4f04-9654-d300a792b9ed","Type":"ContainerDied","Data":"7db841bf4fd8ad28daceb9ce18d8db12d4b4b35396cd7951f020577d79da0574"}
Oct 31 01:09:16.269609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.269219    4553 scope.go:122] "RemoveContainer" containerID="2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"
Oct 31 01:09:16.277649 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.277597    4553 scope.go:122] "RemoveContainer" containerID="695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"
Oct 31 01:09:16.285213 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.285191    4553 scope.go:122] "RemoveContainer" containerID="abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"
Oct 31 01:09:16.292097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292078    4553 scope.go:122] "RemoveContainer" containerID="2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"
Oct 31 01:09:16.292350 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:09:16.292328    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304\": container with ID starting with 2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304 not found: ID does not exist" containerID="2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"
Oct 31 01:09:16.292393 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292361    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304"} err="failed to get container status \"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304\": rpc error: code = NotFound desc = could not find container \"2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304\": container with ID starting with 2f31deb8a5b41dad46523e345fcf3feb3bf94c8386909c335e74da3c42d1a304 not found: ID does not exist"
Oct 31 01:09:16.292393 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292379    4553 scope.go:122] "RemoveContainer" containerID="695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"
Oct 31 01:09:16.292643 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:09:16.292625    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070\": container with ID starting with 695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070 not found: ID does not exist" containerID="695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"
Oct 31 01:09:16.292714 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292649    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070"} err="failed to get container status \"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070\": rpc error: code = NotFound desc = could not find container \"695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070\": container with ID starting with 695b5e1db90e29f87c4f7887fe395ffc53d0e4818ba433c76e61c5caaf217070 not found: ID does not exist"
Oct 31 01:09:16.292714 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292664    4553 scope.go:122] "RemoveContainer" containerID="abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"
Oct 31 01:09:16.292904 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:09:16.292888    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c\": container with ID starting with abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c not found: ID does not exist" containerID="abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"
Oct 31 01:09:16.292948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.292910    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c"} err="failed to get container status \"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c\": rpc error: code = NotFound desc = could not find container \"abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c\": container with ID starting with abfa236f05338348d31265a8085a71bb3b20c5e4f83b0ca1b438ec32fda74a0c not found: ID does not exist"
Oct 31 01:09:16.295123 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.295103    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:09:16.303107 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.303083    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-noslash-predictor-8b8d87978-rtmv2"]
Oct 31 01:09:16.735531 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.735495    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:16.739756 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.739730    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:09:16.740014 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740002    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kube-rbac-proxy"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740016    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kube-rbac-proxy"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740025    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740030    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740049    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kserve-container"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740054    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kserve-container"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740063    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="storage-initializer"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740067    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="storage-initializer"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740074    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="storage-initializer"
Oct 31 01:09:16.740078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740079    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="storage-initializer"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740087    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kube-rbac-proxy"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740091    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kube-rbac-proxy"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740135    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kserve-container"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740143    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kserve-container"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740151    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff34097f-8d9b-416d-84b3-6c89484fd061" containerName="kube-rbac-proxy"
Oct 31 01:09:16.740501 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.740156    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" containerName="kube-rbac-proxy"
Oct 31 01:09:16.744718 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.744699    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.750779 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750736    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:09:16.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750787    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 31 01:09:16.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750833    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 31 01:09:16.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750848    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:09:16.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750856    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:09:16.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750889    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:09:16.751264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.750994    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:09:16.756395 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.756372    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:09:16.837079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.837047    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.837249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.837092    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.837249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.837121    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.837249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.837194    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rtd\" (UniqueName: \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.837249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.837217    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.900114 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.900080    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:16.937782 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.937752    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.937927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.937791    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.937927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.937808    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.937927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.937832    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rtd\" (UniqueName: \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.937927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.937849    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.938230 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.938209    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.938513 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.938490    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.938617 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.938513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.940440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.940412    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:16.952298 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:16.952273    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rtd\" (UniqueName: \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\") pod \"isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:17.062087 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:17.062006    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:17.200196 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:17.200159    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf89c8c-c5fb-4f04-9654-d300a792b9ed" path="/var/lib/kubelet/pods/ebf89c8c-c5fb-4f04-9654-d300a792b9ed/volumes"
Oct 31 01:09:17.340195 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:17.340172    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:09:17.342467 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:09:17.342441    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45a46ff_c503_4271_aaf1_edffb0125a43.slice/crio-91976d8b70a3f08bf2bce809fb43514b6c22783408c58dea069c9b92828a74bb WatchSource:0}: Error finding container 91976d8b70a3f08bf2bce809fb43514b6c22783408c58dea069c9b92828a74bb: Status 404 returned error can't find the container with id 91976d8b70a3f08bf2bce809fb43514b6c22783408c58dea069c9b92828a74bb
Oct 31 01:09:18.283973 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:18.283936    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerStarted","Data":"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"}
Oct 31 01:09:18.284353 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:18.284036    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerStarted","Data":"91976d8b70a3f08bf2bce809fb43514b6c22783408c58dea069c9b92828a74bb"}
Oct 31 01:09:27.129607 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:09:27.129516    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:09:27.317654 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:27.317621    4553 generic.go:356] "Generic (PLEG): container finished" podID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerID="1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a" exitCode=0
Oct 31 01:09:27.317871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:27.317706    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerDied","Data":"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"}
Oct 31 01:09:28.325658 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:28.325625    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerStarted","Data":"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"}
Oct 31 01:09:28.326077 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:28.325703    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerStarted","Data":"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"}
Oct 31 01:09:28.326077 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:28.325907    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:28.355284 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:28.355236    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podStartSLOduration=3.050581523 podStartE2EDuration="12.355219s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.304637477s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:28.354804004 +0000 UTC m=+2431.263806659" watchObservedRunningTime="2026-10-31 01:09:28.355219 +0000 UTC m=+2431.264221655"
Oct 31 01:09:29.330740 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:29.330705    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:29.332052 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:29.332025    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 31 01:09:30.342022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:30.341978    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 31 01:09:35.348366 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:35.348333    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:35.349019 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:35.348993    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 31 01:09:37.587840 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:37.587801    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:42.975592 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:42.975551    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:43.117638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:43.117597    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:45.349884 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:45.349856    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:09:47.252890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:09:47.252856    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:57.130608 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:09:57.130566    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:10:03.523369 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:03.523335    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:10:03.523799 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:03.523564    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container" containerID="cri-o://891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48" gracePeriod=30
Oct 31 01:10:03.523799 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:03.523596    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kube-rbac-proxy" containerID="cri-o://ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c" gracePeriod=30
Oct 31 01:10:03.594615 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:03.594590    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.473357 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:04.473322    4553 generic.go:356] "Generic (PLEG): container finished" podID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerID="ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c" exitCode=0
Oct 31 01:10:04.473570 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:04.473401    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerDied","Data":"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"}
Oct 31 01:10:05.161966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.161944    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:10:05.200142 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200119    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\") pod \"e45a46ff-c503-4271-aaf1-edffb0125a43\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") "
Oct 31 01:10:05.200274 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200161    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\" (UniqueName: \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\") pod \"e45a46ff-c503-4271-aaf1-edffb0125a43\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") "
Oct 31 01:10:05.200274 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200191    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\") pod \"e45a46ff-c503-4271-aaf1-edffb0125a43\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") "
Oct 31 01:10:05.200412 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200293    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\") pod \"e45a46ff-c503-4271-aaf1-edffb0125a43\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") "
Oct 31 01:10:05.200412 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200335    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\") pod \"e45a46ff-c503-4271-aaf1-edffb0125a43\" (UID: \"e45a46ff-c503-4271-aaf1-edffb0125a43\") "
Oct 31 01:10:05.200564 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200528    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config" pod "e45a46ff-c503-4271-aaf1-edffb0125a43" (UID: "e45a46ff-c503-4271-aaf1-edffb0125a43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:05.200770 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200746    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:05.200881 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.200861    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert" pod "e45a46ff-c503-4271-aaf1-edffb0125a43" (UID: "e45a46ff-c503-4271-aaf1-edffb0125a43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:05.201814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.201772    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location" pod "e45a46ff-c503-4271-aaf1-edffb0125a43" (UID: "e45a46ff-c503-4271-aaf1-edffb0125a43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:05.202383 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.202358    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls" pod "e45a46ff-c503-4271-aaf1-edffb0125a43" (UID: "e45a46ff-c503-4271-aaf1-edffb0125a43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:05.202585 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.202571    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd" pod "e45a46ff-c503-4271-aaf1-edffb0125a43" (UID: "e45a46ff-c503-4271-aaf1-edffb0125a43"). InnerVolumeSpecName "kube-api-access-x6rtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:05.302078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.301996    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rtd\" (UniqueName: \"kubernetes.io/projected/e45a46ff-c503-4271-aaf1-edffb0125a43-kube-api-access-x6rtd\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:05.302078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.302027    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e45a46ff-c503-4271-aaf1-edffb0125a43-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:05.302078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.302037    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e45a46ff-c503-4271-aaf1-edffb0125a43-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:05.302078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.302047    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e45a46ff-c503-4271-aaf1-edffb0125a43-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:05.478431 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.478401    4553 generic.go:356] "Generic (PLEG): container finished" podID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerID="891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48" exitCode=0
Oct 31 01:10:05.478584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.478471    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"
Oct 31 01:10:05.478584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.478470    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerDied","Data":"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"}
Oct 31 01:10:05.478660 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.478620    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42" event={"ID":"e45a46ff-c503-4271-aaf1-edffb0125a43","Type":"ContainerDied","Data":"91976d8b70a3f08bf2bce809fb43514b6c22783408c58dea069c9b92828a74bb"}
Oct 31 01:10:05.478660 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.478647    4553 scope.go:122] "RemoveContainer" containerID="ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"
Oct 31 01:10:05.487156 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.487138    4553 scope.go:122] "RemoveContainer" containerID="891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"
Oct 31 01:10:05.494318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.494301    4553 scope.go:122] "RemoveContainer" containerID="1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"
Oct 31 01:10:05.501162 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501144    4553 scope.go:122] "RemoveContainer" containerID="ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"
Oct 31 01:10:05.501419 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:05.501396    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c\": container with ID starting with ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c not found: ID does not exist" containerID="ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"
Oct 31 01:10:05.501493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501431    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c"} err="failed to get container status \"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c\": rpc error: code = NotFound desc = could not find container \"ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c\": container with ID starting with ef33dd18017b6ce8a9a3de9a4b149ae90b8fe585e0681a6b80758bf4477d865c not found: ID does not exist"
Oct 31 01:10:05.501493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501453    4553 scope.go:122] "RemoveContainer" containerID="891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"
Oct 31 01:10:05.501714 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:05.501698    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48\": container with ID starting with 891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48 not found: ID does not exist" containerID="891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"
Oct 31 01:10:05.501764 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501721    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48"} err="failed to get container status \"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48\": rpc error: code = NotFound desc = could not find container \"891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48\": container with ID starting with 891fb50118bc199a0972477d712826030aebbf4e20289eee9afcacd7989f4d48 not found: ID does not exist"
Oct 31 01:10:05.501764 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501734    4553 scope.go:122] "RemoveContainer" containerID="1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"
Oct 31 01:10:05.501935 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:05.501924    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a\": container with ID starting with 1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a not found: ID does not exist" containerID="1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"
Oct 31 01:10:05.501972 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.501938    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a"} err="failed to get container status \"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a\": rpc error: code = NotFound desc = could not find container \"1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a\": container with ID starting with 1ed10b7d02eb33db186e094f184b96d37fbfe1158a38d0461b1df7495c582e5a not found: ID does not exist"
Oct 31 01:10:05.504412 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.504393    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:10:05.512733 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:05.512714    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-predictor-6dc6d57-xmj42"]
Oct 31 01:10:06.032311 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032279    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:06.032566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032555    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="storage-initializer"
Oct 31 01:10:06.032610 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032567    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="storage-initializer"
Oct 31 01:10:06.032610 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032583    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container"
Oct 31 01:10:06.032610 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032588    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container"
Oct 31 01:10:06.032610 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032595    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kube-rbac-proxy"
Oct 31 01:10:06.032610 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032599    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kube-rbac-proxy"
Oct 31 01:10:06.032793 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032643    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kube-rbac-proxy"
Oct 31 01:10:06.032793 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.032655    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" containerName="kserve-container"
Oct 31 01:10:06.037484 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.037463    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.040045 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.040020    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:06.043590 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.043566    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 31 01:10:06.043744 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.043590    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:10:06.043744 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.043590    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:10:06.043744 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.043544    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:10:06.043744 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.043625    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:10:06.044257 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.044229    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:10:06.044324 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.044281    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 31 01:10:06.047961 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.047940    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:06.109604 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.109571    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ftrq\" (UniqueName: \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.109777 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.109611    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.109777 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.109706    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.109777 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.109730    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.109777 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.109751    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.189572 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.189539    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:06.211107 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211076    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ftrq\" (UniqueName: \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211113    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211141    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211161    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211283    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211702 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211655    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211926 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211907    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.211996 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.211940    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.213688 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.213662    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.225971 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.225944    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ftrq\" (UniqueName: \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.354638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.354577    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:06.634913 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:06.634889    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:06.637056 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:10:06.637029    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e767dcd_c75a_43e1_9fd5_28e7483b654d.slice/crio-2e4b23c89df3bfef00a21c6c328bd66ff244e29061fc530742e494ee9b7e4f9d WatchSource:0}: Error finding container 2e4b23c89df3bfef00a21c6c328bd66ff244e29061fc530742e494ee9b7e4f9d: Status 404 returned error can't find the container with id 2e4b23c89df3bfef00a21c6c328bd66ff244e29061fc530742e494ee9b7e4f9d
Oct 31 01:10:07.200194 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:07.200159    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45a46ff-c503-4271-aaf1-edffb0125a43" path="/var/lib/kubelet/pods/e45a46ff-c503-4271-aaf1-edffb0125a43/volumes"
Oct 31 01:10:07.488110 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:07.488022    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerStarted","Data":"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"}
Oct 31 01:10:07.488110 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:07.488075    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerStarted","Data":"2e4b23c89df3bfef00a21c6c328bd66ff244e29061fc530742e494ee9b7e4f9d"}
Oct 31 01:10:11.510139 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:11.510108    4553 generic.go:356] "Generic (PLEG): container finished" podID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerID="2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015" exitCode=0
Oct 31 01:10:11.510564 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:11.510189    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerDied","Data":"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"}
Oct 31 01:10:12.517328 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.517293    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerStarted","Data":"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"}
Oct 31 01:10:12.517782 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.517355    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerStarted","Data":"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"}
Oct 31 01:10:12.517782 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.517705    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:12.517917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.517873    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:12.519369 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.519344    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 31 01:10:12.548552 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:12.548495    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podStartSLOduration=1.817200711 podStartE2EDuration="6.548477472s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.731276761s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:12.547209197 +0000 UTC m=+2475.456211851" watchObservedRunningTime="2026-10-31 01:10:12.548477472 +0000 UTC m=+2475.457480126"
Oct 31 01:10:13.521540 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:13.521503    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 31 01:10:14.525407 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:14.525312    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 31 01:10:19.529764 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:19.529736    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:19.530248 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:19.530217    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 31 01:10:25.468951 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:25.468910    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:27.129568 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:27.129520    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:10:29.530851 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:29.530825    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:33.306807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:33.306765    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:40.724739 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:40.724665    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:40.875739 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:40.875691    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:41.488421 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.488384    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:41.488688 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.488636    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container" containerID="cri-o://e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7" gracePeriod=30
Oct 31 01:10:41.488758 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.488710    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kube-rbac-proxy" containerID="cri-o://415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873" gracePeriod=30
Oct 31 01:10:41.564858 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.564825    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:41.636850 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.636822    4553 generic.go:356] "Generic (PLEG): container finished" podID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerID="415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873" exitCode=0
Oct 31 01:10:41.636957 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:41.636857    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerDied","Data":"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"}
Oct 31 01:10:42.820242 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.820221    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:42.993527 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.993454    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\") pod \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") "
Oct 31 01:10:42.993527 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.993501    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\") pod \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") "
Oct 31 01:10:42.993816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.993547    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\") pod \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") "
Oct 31 01:10:42.993816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.993574    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\" (UniqueName: \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\") pod \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") "
Oct 31 01:10:42.993816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.993612    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\") pod \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\" (UID: \"3e767dcd-c75a-43e1-9fd5-28e7483b654d\") "
Oct 31 01:10:42.994117 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.994078    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config" pod "3e767dcd-c75a-43e1-9fd5-28e7483b654d" (UID: "3e767dcd-c75a-43e1-9fd5-28e7483b654d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:42.994117 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.994075    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert" pod "3e767dcd-c75a-43e1-9fd5-28e7483b654d" (UID: "3e767dcd-c75a-43e1-9fd5-28e7483b654d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:42.995023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.994993    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location" pod "3e767dcd-c75a-43e1-9fd5-28e7483b654d" (UID: "3e767dcd-c75a-43e1-9fd5-28e7483b654d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:42.996037 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.996016    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls" pod "3e767dcd-c75a-43e1-9fd5-28e7483b654d" (UID: "3e767dcd-c75a-43e1-9fd5-28e7483b654d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:42.996112 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:42.996062    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq" pod "3e767dcd-c75a-43e1-9fd5-28e7483b654d" (UID: "3e767dcd-c75a-43e1-9fd5-28e7483b654d"). InnerVolumeSpecName "kube-api-access-9ftrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:43.094755 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.094731    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:43.094755 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.094755    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:43.094904 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.094765    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ftrq\" (UniqueName: \"kubernetes.io/projected/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kube-api-access-9ftrq\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:43.094904 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.094774    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e767dcd-c75a-43e1-9fd5-28e7483b654d-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:43.094904 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.094783    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e767dcd-c75a-43e1-9fd5-28e7483b654d-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:43.646631 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.646590    4553 generic.go:356] "Generic (PLEG): container finished" podID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerID="e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7" exitCode=0
Oct 31 01:10:43.646823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.646635    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerDied","Data":"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"}
Oct 31 01:10:43.646823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.646655    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"
Oct 31 01:10:43.646823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.646715    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86" event={"ID":"3e767dcd-c75a-43e1-9fd5-28e7483b654d","Type":"ContainerDied","Data":"2e4b23c89df3bfef00a21c6c328bd66ff244e29061fc530742e494ee9b7e4f9d"}
Oct 31 01:10:43.646823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.646756    4553 scope.go:122] "RemoveContainer" containerID="415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"
Oct 31 01:10:43.654891 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.654873    4553 scope.go:122] "RemoveContainer" containerID="e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"
Oct 31 01:10:43.661994 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.661972    4553 scope.go:122] "RemoveContainer" containerID="2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"
Oct 31 01:10:43.669035 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669018    4553 scope.go:122] "RemoveContainer" containerID="415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"
Oct 31 01:10:43.669280 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:43.669263    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873\": container with ID starting with 415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873 not found: ID does not exist" containerID="415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"
Oct 31 01:10:43.669336 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669292    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873"} err="failed to get container status \"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873\": rpc error: code = NotFound desc = could not find container \"415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873\": container with ID starting with 415b0525e624d751be04a5e555c5a25eddf59a55b6b0fbc90f148ee8f6ba1873 not found: ID does not exist"
Oct 31 01:10:43.669336 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669308    4553 scope.go:122] "RemoveContainer" containerID="e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"
Oct 31 01:10:43.669535 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:43.669520    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7\": container with ID starting with e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7 not found: ID does not exist" containerID="e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"
Oct 31 01:10:43.669584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669538    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7"} err="failed to get container status \"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7\": rpc error: code = NotFound desc = could not find container \"e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7\": container with ID starting with e9281f8f1714bb2b0ba9b7d2f928fa4a68996dbe529d66c3866f461b125cf7b7 not found: ID does not exist"
Oct 31 01:10:43.669584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669549    4553 scope.go:122] "RemoveContainer" containerID="2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"
Oct 31 01:10:43.669821 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:43.669802    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015\": container with ID starting with 2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015 not found: ID does not exist" containerID="2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"
Oct 31 01:10:43.669865 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.669830    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015"} err="failed to get container status \"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015\": rpc error: code = NotFound desc = could not find container \"2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015\": container with ID starting with 2267454d7191c5d08717aa3192a04ac21851fd2ed60011395fb34a58ac369015 not found: ID does not exist"
Oct 31 01:10:43.672539 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.672518    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:43.678563 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:43.678544    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-ts-v1-noslash-predictor-57b8b5bb46-ljw86"]
Oct 31 01:10:45.199503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:45.199463    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" path="/var/lib/kubelet/pods/3e767dcd-c75a-43e1-9fd5-28e7483b654d/volumes"
Oct 31 01:10:46.036702 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.036655    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:46.150997 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.150960    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:10:46.151325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151308    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kube-rbac-proxy"
Oct 31 01:10:46.151409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151327    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kube-rbac-proxy"
Oct 31 01:10:46.151409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151343    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="storage-initializer"
Oct 31 01:10:46.151409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151350    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="storage-initializer"
Oct 31 01:10:46.151409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151374    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container"
Oct 31 01:10:46.151409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151382    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container"
Oct 31 01:10:46.151664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151472    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kube-rbac-proxy"
Oct 31 01:10:46.151664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.151487    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e767dcd-c75a-43e1-9fd5-28e7483b654d" containerName="kserve-container"
Oct 31 01:10:46.156312 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.156292    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.163351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.163328    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:10:46.163351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.163334    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:10:46.163351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.163334    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 31 01:10:46.163553 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.163383    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:10:46.164252 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.164234    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:10:46.164344 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.164274    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 31 01:10:46.164344 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.164294    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:10:46.166580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.166560    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:10:46.204093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.204066    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:46.318853 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.318791    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.318853 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.318851    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.319108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.319089    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.319186 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.319171    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.319258 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.319243    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4s2q\" (UniqueName: \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.380155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.380133    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:46.420154 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420123    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420161    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420188    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4s2q\" (UniqueName: \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420244    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420587 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420567    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420848 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420830    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.420919 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.420903    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.422786 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.422771    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.434919 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.434894    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4s2q\" (UniqueName: \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\") pod \"isvc-canary-create-predictor-74fddf7499-nfpm2\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.473595 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.473573    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:10:46.750749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:46.750723    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:10:46.753009 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:10:46.752978    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ceb918_f62e_41ad_b608_f2db3b56cdd8.slice/crio-e0212930524e63c57a809f178b47c62454bbe597897a30283b11a39b1b6cd8a1 WatchSource:0}: Error finding container e0212930524e63c57a809f178b47c62454bbe597897a30283b11a39b1b6cd8a1: Status 404 returned error can't find the container with id e0212930524e63c57a809f178b47c62454bbe597897a30283b11a39b1b6cd8a1
Oct 31 01:10:47.312701 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:47.312649    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:47.663594 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:47.663558    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerStarted","Data":"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"}
Oct 31 01:10:47.663774 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:47.663628    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerStarted","Data":"e0212930524e63c57a809f178b47c62454bbe597897a30283b11a39b1b6cd8a1"}
Oct 31 01:10:51.681207 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:51.681172    4553 generic.go:356] "Generic (PLEG): container finished" podID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerID="cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65" exitCode=0
Oct 31 01:10:51.681596 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:51.681216    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerDied","Data":"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"}
Oct 31 01:10:51.682343 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:10:51.682326    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:10:57.130945 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:10:57.130886    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:11:05.746267 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:05.746225    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerStarted","Data":"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"}
Oct 31 01:11:05.746656 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:05.746299    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerStarted","Data":"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"}
Oct 31 01:11:05.746656 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:05.746501    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:11:05.780213 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:05.780156    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podStartSLOduration=2.3661594040000002 podStartE2EDuration="19.780138943s" totalImagesPullingTime="13.565182801s" totalInitContainerRuntime="3.848796738s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:05.779222544 +0000 UTC m=+2528.688225198" watchObservedRunningTime="2026-10-31 01:11:05.780138943 +0000 UTC m=+2528.689141597"
Oct 31 01:11:06.750117 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:06.750083    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:11:06.751514 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:06.751487    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:07.753961 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:07.753917    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:12.758345 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:12.758315    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:11:12.758925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:12.758894    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:22.759114 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:22.759074    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:27.129839 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:11:27.129805    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:11:31.121000 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:31.120961    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:32.759039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:32.759002    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:40.508738 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.508620    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:40.540528 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.540499    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:11:40.573756 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.573721    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:11:40.573897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.573768    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.582073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.582045    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 31 01:11:40.582213 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.582079    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:11:40.582213 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.582110    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:11:40.582213 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.582119    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:11:40.582914 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.582896    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:11:40.583169 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.583155    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 31 01:11:40.583247 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.583212    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:11:40.653159 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.653125    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:40.751478 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.751442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.751614 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.751483    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvcm\" (UniqueName: \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.751614 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.751499    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.751614 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.751553    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.751614 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.751579    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.852956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.852878    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.852956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.852923    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvcm\" (UniqueName: \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.852956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.852941    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.853226 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.852965    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.853226 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.852990    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.853513 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.853490    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.853741 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.853723    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.853834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.853760    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.855524 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.855497    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.867886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.867863    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvcm\" (UniqueName: \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\") pod \"isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:40.891388 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:40.891359    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:11:41.167880 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:41.167858    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:11:41.170389 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:11:41.170362    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358cc6ca_9a9b_404c_b428_7ee806519e42.slice/crio-dd0ae5b1a893505df6de1ba8ae9107002b6ec4b3ecd7f83dc378b01fab596880 WatchSource:0}: Error finding container dd0ae5b1a893505df6de1ba8ae9107002b6ec4b3ecd7f83dc378b01fab596880: Status 404 returned error can't find the container with id dd0ae5b1a893505df6de1ba8ae9107002b6ec4b3ecd7f83dc378b01fab596880
Oct 31 01:11:41.880956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:41.880923    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerStarted","Data":"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"}
Oct 31 01:11:41.881488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:41.880979    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerStarted","Data":"dd0ae5b1a893505df6de1ba8ae9107002b6ec4b3ecd7f83dc378b01fab596880"}
Oct 31 01:11:42.759428 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:42.759391    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:45.898017 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:45.897980    4553 generic.go:356] "Generic (PLEG): container finished" podID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerID="28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1" exitCode=0
Oct 31 01:11:45.898457 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:45.898054    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerDied","Data":"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"}
Oct 31 01:11:47.360840 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:47.360804    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:52.759743 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:11:52.759698    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:11:57.132654 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:11:57.132607    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:12:02.759063 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:02.759020    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:12:03.983331 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:03.983289    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerStarted","Data":"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"}
Oct 31 01:12:03.983733 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:03.983384    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerStarted","Data":"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"}
Oct 31 01:12:03.983733 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:03.983593    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:12:04.013088 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:04.013044    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podStartSLOduration=2.395252338 podStartE2EDuration="24.013033065s" totalImagesPullingTime="17.730343414s" totalInitContainerRuntime="3.887437313s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:04.011192008 +0000 UTC m=+2586.920194662" watchObservedRunningTime="2026-10-31 01:12:04.013033065 +0000 UTC m=+2586.922035719"
Oct 31 01:12:04.987592 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:04.987558    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:12:04.988564 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:04.988533    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:05.991909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:05.991868    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:10.996602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:10.996566    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:12:10.997191 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:10.997165    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:12.759437 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:12.759398    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:12:18.961479 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:18.961438    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:20.997962 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:20.997919    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:21.194955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:21.194917    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Oct 31 01:12:27.130666 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:12:27.130628    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:12:30.997149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:30.997109    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:31.199216 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:31.199187    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:12:36.365432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:36.365393    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:12:36.365942 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:36.365636    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container" containerID="cri-o://9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b" gracePeriod=30
Oct 31 01:12:36.365942 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:36.365713    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kube-rbac-proxy" containerID="cri-o://b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704" gracePeriod=30
Oct 31 01:12:36.433093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:36.433069    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:36.577896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:36.577862    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:37.117548 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:37.117515    4553 generic.go:356] "Generic (PLEG): container finished" podID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerID="b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704" exitCode=0
Oct 31 01:12:37.117741 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:37.117585    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerDied","Data":"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"}
Oct 31 01:12:37.755059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:37.755014    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Oct 31 01:12:40.997163 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:40.997128    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:41.004219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.004196    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:12:41.135217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.135130    4553 generic.go:356] "Generic (PLEG): container finished" podID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerID="9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b" exitCode=0
Oct 31 01:12:41.135217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.135167    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerDied","Data":"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"}
Oct 31 01:12:41.135217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.135184    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"
Oct 31 01:12:41.135437 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.135243    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2" event={"ID":"a6ceb918-f62e-41ad-b608-f2db3b56cdd8","Type":"ContainerDied","Data":"e0212930524e63c57a809f178b47c62454bbe597897a30283b11a39b1b6cd8a1"}
Oct 31 01:12:41.135437 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.135272    4553 scope.go:122] "RemoveContainer" containerID="b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"
Oct 31 01:12:41.142877 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.142854    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\") pod \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") "
Oct 31 01:12:41.143024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.142907    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\") pod \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") "
Oct 31 01:12:41.143024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.142941    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\" (UniqueName: \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\") pod \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") "
Oct 31 01:12:41.143024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.142980    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\") pod \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") "
Oct 31 01:12:41.143182 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.143033    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\") pod \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\" (UID: \"a6ceb918-f62e-41ad-b608-f2db3b56cdd8\") "
Oct 31 01:12:41.143301 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.143244    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert" pod "a6ceb918-f62e-41ad-b608-f2db3b56cdd8" (UID: "a6ceb918-f62e-41ad-b608-f2db3b56cdd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:12:41.143361 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.143325    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config" pod "a6ceb918-f62e-41ad-b608-f2db3b56cdd8" (UID: "a6ceb918-f62e-41ad-b608-f2db3b56cdd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:12:41.143515 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.143497    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location" pod "a6ceb918-f62e-41ad-b608-f2db3b56cdd8" (UID: "a6ceb918-f62e-41ad-b608-f2db3b56cdd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:12:41.143619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.143559    4553 scope.go:122] "RemoveContainer" containerID="9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"
Oct 31 01:12:41.145368 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.145345    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls" pod "a6ceb918-f62e-41ad-b608-f2db3b56cdd8" (UID: "a6ceb918-f62e-41ad-b608-f2db3b56cdd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:12:41.145452 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.145434    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q" pod "a6ceb918-f62e-41ad-b608-f2db3b56cdd8" (UID: "a6ceb918-f62e-41ad-b608-f2db3b56cdd8"). InnerVolumeSpecName "kube-api-access-k4s2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:41.161615 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.161591    4553 scope.go:122] "RemoveContainer" containerID="cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"
Oct 31 01:12:41.168706 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.168668    4553 scope.go:122] "RemoveContainer" containerID="b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"
Oct 31 01:12:41.168987 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:12:41.168966    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704\": container with ID starting with b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704 not found: ID does not exist" containerID="b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"
Oct 31 01:12:41.169037 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.168994    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704"} err="failed to get container status \"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704\": rpc error: code = NotFound desc = could not find container \"b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704\": container with ID starting with b2244759992464d654739664c44b4b6c4b0364b788eb36cc1f1c05293dff0704 not found: ID does not exist"
Oct 31 01:12:41.169037 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.169011    4553 scope.go:122] "RemoveContainer" containerID="9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"
Oct 31 01:12:41.169229 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:12:41.169215    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b\": container with ID starting with 9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b not found: ID does not exist" containerID="9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"
Oct 31 01:12:41.169276 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.169233    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b"} err="failed to get container status \"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b\": rpc error: code = NotFound desc = could not find container \"9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b\": container with ID starting with 9715ec0626b9ba3d325c8b8146416e018ca92d1584d557eb6a1a85856c529d1b not found: ID does not exist"
Oct 31 01:12:41.169276 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.169245    4553 scope.go:122] "RemoveContainer" containerID="cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"
Oct 31 01:12:41.169437 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:12:41.169419    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65\": container with ID starting with cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65 not found: ID does not exist" containerID="cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"
Oct 31 01:12:41.169474 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.169441    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65"} err="failed to get container status \"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65\": rpc error: code = NotFound desc = could not find container \"cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65\": container with ID starting with cf8c5f467040f0cb25089fe7c1704df1486d043d95c4a68b894abb40bd255c65 not found: ID does not exist"
Oct 31 01:12:41.243989 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.243966    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:12:41.243989 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.243987    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:12:41.244108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.243996    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:12:41.244108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.244006    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4s2q\" (UniqueName: \"kubernetes.io/projected/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-kube-api-access-k4s2q\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:12:41.244108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.244014    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ceb918-f62e-41ad-b608-f2db3b56cdd8-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:12:41.455379 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.455348    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:12:41.461352 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:41.461327    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-nfpm2"]
Oct 31 01:12:42.939780 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.939743    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940038    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kube-rbac-proxy"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940049    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kube-rbac-proxy"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940066    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="storage-initializer"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940072    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="storage-initializer"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940088    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940093    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container"
Oct 31 01:12:42.940148 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940143    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kserve-container"
Oct 31 01:12:42.940368 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.940152    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" containerName="kube-rbac-proxy"
Oct 31 01:12:42.945076 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.945056    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:42.953337 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.952262    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:42.958073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957419    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:12:42.958073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957433    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:12:42.958073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957452    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 31 01:12:42.958073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957459    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:12:42.958073 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957511    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:12:42.958406 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.957712    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:12:42.958595 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.958575    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 31 01:12:42.960476 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:42.960453    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:12:43.058907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.058872    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.059079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.058928    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.059079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.058997    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.059079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.059074    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.059181 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.059101    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdl7\" (UniqueName: \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.065202 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.065179    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:12:43.068490 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.068476    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.081542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.081517    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:12:43.099173 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.099151    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:43.155904 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.155877    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 31 01:12:43.160189 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160169    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdl7\" (UniqueName: \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.160251 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.160251 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160222    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mp9\" (UniqueName: \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.160335 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160251    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.160385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160368    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.160421 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160393    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.160460 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160432    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.160502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160462    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.160597 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160570    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.160796 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160609    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.160902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.160881    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.161059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.161039    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.161126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.161091    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.163115 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.163093    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.174954 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.174935    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdl7\" (UniqueName: \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-899l7\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.199343 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.199288    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ceb918-f62e-41ad-b608-f2db3b56cdd8" path="/var/lib/kubelet/pods/a6ceb918-f62e-41ad-b608-f2db3b56cdd8/volumes"
Oct 31 01:12:43.243412 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.243387    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:43.262279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.262240    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.262436 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.262417    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mp9\" (UniqueName: \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.262568 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.262554    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.262724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.262708    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.262914 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.262900    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.263355 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.263336    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.264091 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.264070    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.267376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.267355    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:43.267871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.267848    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.267965 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.267354    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.279544 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.279519    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mp9\" (UniqueName: \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\") pod \"isvc-canary-promote-predictor-59b8558b77-qr94z\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.386305 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.386269    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:43.565666 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.565642    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:12:43.567693 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:12:43.567657    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea0f89d_b9c1_421b_9512_27ebd06334ef.slice/crio-51c0c357c8e5dab5ad8ece07fa5d287afeafb31079339a31844a8515240a4a26 WatchSource:0}: Error finding container 51c0c357c8e5dab5ad8ece07fa5d287afeafb31079339a31844a8515240a4a26: Status 404 returned error can't find the container with id 51c0c357c8e5dab5ad8ece07fa5d287afeafb31079339a31844a8515240a4a26
Oct 31 01:12:43.664264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:43.664237    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:12:43.666377 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:12:43.666351    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe428c6_9f71_4af6_a72b_0b456f1306ed.slice/crio-b8007f954429e4198d3f01267c11b99359173df58546325467fe2052b95fd8ec WatchSource:0}: Error finding container b8007f954429e4198d3f01267c11b99359173df58546325467fe2052b95fd8ec: Status 404 returned error can't find the container with id b8007f954429e4198d3f01267c11b99359173df58546325467fe2052b95fd8ec
Oct 31 01:12:44.150376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:44.150330    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerStarted","Data":"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"}
Oct 31 01:12:44.151032 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:44.150404    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerStarted","Data":"51c0c357c8e5dab5ad8ece07fa5d287afeafb31079339a31844a8515240a4a26"}
Oct 31 01:12:44.152116 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:44.152090    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerStarted","Data":"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"}
Oct 31 01:12:44.152249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:44.152135    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerStarted","Data":"b8007f954429e4198d3f01267c11b99359173df58546325467fe2052b95fd8ec"}
Oct 31 01:12:47.417650 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:47.417619    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:48.168029 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:48.168000    4553 generic.go:356] "Generic (PLEG): container finished" podID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerID="dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba" exitCode=0
Oct 31 01:12:48.168208 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:48.168082    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerDied","Data":"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"}
Oct 31 01:12:48.169579 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:48.169558    4553 generic.go:356] "Generic (PLEG): container finished" podID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerID="a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d" exitCode=0
Oct 31 01:12:48.169702 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:48.169622    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerDied","Data":"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"}
Oct 31 01:12:49.181308 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.181261    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerStarted","Data":"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"}
Oct 31 01:12:49.181783 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.181377    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerStarted","Data":"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"}
Oct 31 01:12:49.181783 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.181626    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:49.183523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.183496    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerStarted","Data":"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"}
Oct 31 01:12:49.183657 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.183538    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerStarted","Data":"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"}
Oct 31 01:12:49.183799 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.183782    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:49.183867 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.183821    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:49.185036 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.185016    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:12:49.209875 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.209822    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podStartSLOduration=2.30197333 podStartE2EDuration="6.209810129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.907836799s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:49.208717368 +0000 UTC m=+2632.117720023" watchObservedRunningTime="2026-10-31 01:12:49.209810129 +0000 UTC m=+2632.118812783"
Oct 31 01:12:49.235201 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:49.235153    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podStartSLOduration=3.235776055 podStartE2EDuration="7.23513986s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.999363805s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:49.234419287 +0000 UTC m=+2632.143421941" watchObservedRunningTime="2026-10-31 01:12:49.23513986 +0000 UTC m=+2632.144142513"
Oct 31 01:12:50.188467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:50.188423    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:12:50.189100 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:50.188640    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:50.189992 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:50.189969    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:12:50.997260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:50.997221    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:12:51.192147 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:51.192111    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:12:55.193563 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:55.193530    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:12:55.194027 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:55.194000    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:12:56.196535 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:56.196505    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:12:56.196964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:12:56.196940    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:12:57.130291 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:12:57.130248    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:13:00.997617 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:00.997581    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 31 01:13:05.194026 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:05.193991    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:06.196917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:06.196879    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:10.997862 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:10.997794    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:13:15.194495 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:15.194458    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:16.197009 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:16.196972    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:21.434926 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:21.434887    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:13:21.435496 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:21.435173    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container" containerID="cri-o://2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0" gracePeriod=30
Oct 31 01:13:21.435496 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:21.435255    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kube-rbac-proxy" containerID="cri-o://e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e" gracePeriod=30
Oct 31 01:13:21.503709 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:21.503667    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:22.318872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:22.318839    4553 generic.go:356] "Generic (PLEG): container finished" podID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerID="e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e" exitCode=0
Oct 31 01:13:22.318872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:22.318870    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerDied","Data":"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"}
Oct 31 01:13:25.194553 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.194512    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:25.480813 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.480790    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:13:25.607007 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.606977    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\" (UniqueName: \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\") pod \"358cc6ca-9a9b-404c-b428-7ee806519e42\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") "
Oct 31 01:13:25.607007 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607021    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\" (UniqueName: \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\") pod \"358cc6ca-9a9b-404c-b428-7ee806519e42\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") "
Oct 31 01:13:25.607259 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607048    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\") pod \"358cc6ca-9a9b-404c-b428-7ee806519e42\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") "
Oct 31 01:13:25.607259 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607112    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\") pod \"358cc6ca-9a9b-404c-b428-7ee806519e42\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") "
Oct 31 01:13:25.607259 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607143    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\") pod \"358cc6ca-9a9b-404c-b428-7ee806519e42\" (UID: \"358cc6ca-9a9b-404c-b428-7ee806519e42\") "
Oct 31 01:13:25.607481 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607449    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert" pod "358cc6ca-9a9b-404c-b428-7ee806519e42" (UID: "358cc6ca-9a9b-404c-b428-7ee806519e42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:25.607533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607457    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location" pod "358cc6ca-9a9b-404c-b428-7ee806519e42" (UID: "358cc6ca-9a9b-404c-b428-7ee806519e42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:25.607533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.607517    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config" pod "358cc6ca-9a9b-404c-b428-7ee806519e42" (UID: "358cc6ca-9a9b-404c-b428-7ee806519e42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:25.609320 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.609298    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls" pod "358cc6ca-9a9b-404c-b428-7ee806519e42" (UID: "358cc6ca-9a9b-404c-b428-7ee806519e42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:13:25.609477 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.609458    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm" pod "358cc6ca-9a9b-404c-b428-7ee806519e42" (UID: "358cc6ca-9a9b-404c-b428-7ee806519e42"). InnerVolumeSpecName "kube-api-access-tgvcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:25.708126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.708093    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358cc6ca-9a9b-404c-b428-7ee806519e42-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:25.708126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.708120    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:25.708126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.708130    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358cc6ca-9a9b-404c-b428-7ee806519e42-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:25.708333 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.708139    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvcm\" (UniqueName: \"kubernetes.io/projected/358cc6ca-9a9b-404c-b428-7ee806519e42-kube-api-access-tgvcm\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:25.708333 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:25.708148    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/358cc6ca-9a9b-404c-b428-7ee806519e42-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:26.197907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.197864    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:26.336807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.336773    4553 generic.go:356] "Generic (PLEG): container finished" podID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerID="2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0" exitCode=0
Oct 31 01:13:26.336980 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.336828    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"
Oct 31 01:13:26.336980 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.336830    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerDied","Data":"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"}
Oct 31 01:13:26.336980 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.336943    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm" event={"ID":"358cc6ca-9a9b-404c-b428-7ee806519e42","Type":"ContainerDied","Data":"dd0ae5b1a893505df6de1ba8ae9107002b6ec4b3ecd7f83dc378b01fab596880"}
Oct 31 01:13:26.336980 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.336976    4553 scope.go:122] "RemoveContainer" containerID="e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"
Oct 31 01:13:26.345712 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.345672    4553 scope.go:122] "RemoveContainer" containerID="2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"
Oct 31 01:13:26.353396 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.353208    4553 scope.go:122] "RemoveContainer" containerID="28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"
Oct 31 01:13:26.360705 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.360655    4553 scope.go:122] "RemoveContainer" containerID="e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"
Oct 31 01:13:26.361003 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:13:26.360983    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e\": container with ID starting with e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e not found: ID does not exist" containerID="e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"
Oct 31 01:13:26.361055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.361013    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e"} err="failed to get container status \"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e\": rpc error: code = NotFound desc = could not find container \"e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e\": container with ID starting with e8e1f356afb282d8dd756305664c5bfb8e489ee76df01952974bc41a0592e28e not found: ID does not exist"
Oct 31 01:13:26.361055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.361029    4553 scope.go:122] "RemoveContainer" containerID="2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"
Oct 31 01:13:26.361228 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:13:26.361213    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0\": container with ID starting with 2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0 not found: ID does not exist" containerID="2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"
Oct 31 01:13:26.361275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.361232    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0"} err="failed to get container status \"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0\": rpc error: code = NotFound desc = could not find container \"2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0\": container with ID starting with 2b88b80386a39870da105de7f2f90d041c81a6ec7d9badd4d21db2a4346aabf0 not found: ID does not exist"
Oct 31 01:13:26.361275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.361243    4553 scope.go:122] "RemoveContainer" containerID="28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"
Oct 31 01:13:26.361528 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:13:26.361505    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1\": container with ID starting with 28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1 not found: ID does not exist" containerID="28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"
Oct 31 01:13:26.361591 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.361535    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1"} err="failed to get container status \"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1\": rpc error: code = NotFound desc = could not find container \"28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1\": container with ID starting with 28e7ef05e434983d49b4837fbe121ea28ec212521c5c07884db56571f8b2ddc1 not found: ID does not exist"
Oct 31 01:13:26.363761 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.363737    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:13:26.369857 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.369834    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-runtime-predictor-6b8c59fc89-zvhvm"]
Oct 31 01:13:26.924806 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:26.924771    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:27.090691 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:27.090643    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:27.129055 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:13:27.129016    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:13:27.199634 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:27.199561    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" path="/var/lib/kubelet/pods/358cc6ca-9a9b-404c-b428-7ee806519e42/volumes"
Oct 31 01:13:35.194023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:35.193977    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:36.197393 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:36.197355    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:45.194937 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:45.194894    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:46.196962 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:46.196920    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:47.476155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:47.476121    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:55.194405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:55.194358    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:13:56.197729 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:56.197663    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:13:57.129721 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:13:57.129665    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:13:57.401275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:57.401250    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:13:57.406525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:13:57.406506    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:14:05.194611 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:05.194582    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:14:06.197772 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:06.197738    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:14:13.225850 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.225812    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:14:13.226327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.226100    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" containerID="cri-o://83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686" gracePeriod=30
Oct 31 01:14:13.226327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.226171    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kube-rbac-proxy" containerID="cri-o://3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a" gracePeriod=30
Oct 31 01:14:13.319644 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.319618    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:13.527659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.527572    4553 generic.go:356] "Generic (PLEG): container finished" podID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerID="3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a" exitCode=0
Oct 31 01:14:13.527659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:13.527639    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerDied","Data":"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"}
Oct 31 01:14:16.193064 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:16.193019    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Oct 31 01:14:16.197655 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:16.197632    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 31 01:14:17.857511 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.857489    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:14:17.895894 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.895865    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\") pod \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") "
Oct 31 01:14:17.896049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.895908    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\" (UniqueName: \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\") pod \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") "
Oct 31 01:14:17.896049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.895947    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\") pod \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") "
Oct 31 01:14:17.896049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.895965    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\") pod \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") "
Oct 31 01:14:17.896049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.895997    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\") pod \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\" (UID: \"cbe428c6-9f71-4af6-a72b-0b456f1306ed\") "
Oct 31 01:14:17.896326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.896296    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location" pod "cbe428c6-9f71-4af6-a72b-0b456f1306ed" (UID: "cbe428c6-9f71-4af6-a72b-0b456f1306ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:17.896389 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.896236    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert" pod "cbe428c6-9f71-4af6-a72b-0b456f1306ed" (UID: "cbe428c6-9f71-4af6-a72b-0b456f1306ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:17.896447 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.896386    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config" pod "cbe428c6-9f71-4af6-a72b-0b456f1306ed" (UID: "cbe428c6-9f71-4af6-a72b-0b456f1306ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:17.898243 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.898187    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9" pod "cbe428c6-9f71-4af6-a72b-0b456f1306ed" (UID: "cbe428c6-9f71-4af6-a72b-0b456f1306ed"). InnerVolumeSpecName "kube-api-access-27mp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:17.898243 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.898212    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls" pod "cbe428c6-9f71-4af6-a72b-0b456f1306ed" (UID: "cbe428c6-9f71-4af6-a72b-0b456f1306ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:14:17.997318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.997295    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:17.997318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.997317    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe428c6-9f71-4af6-a72b-0b456f1306ed-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:17.997452 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.997327    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mp9\" (UniqueName: \"kubernetes.io/projected/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kube-api-access-27mp9\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:17.997452 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.997336    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbe428c6-9f71-4af6-a72b-0b456f1306ed-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:17.997452 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:17.997344    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe428c6-9f71-4af6-a72b-0b456f1306ed-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:18.548299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.548259    4553 generic.go:356] "Generic (PLEG): container finished" podID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerID="83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686" exitCode=0
Oct 31 01:14:18.548462 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.548329    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerDied","Data":"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"}
Oct 31 01:14:18.548462 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.548343    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"
Oct 31 01:14:18.548462 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.548419    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z" event={"ID":"cbe428c6-9f71-4af6-a72b-0b456f1306ed","Type":"ContainerDied","Data":"b8007f954429e4198d3f01267c11b99359173df58546325467fe2052b95fd8ec"}
Oct 31 01:14:18.548462 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.548457    4553 scope.go:122] "RemoveContainer" containerID="3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"
Oct 31 01:14:18.557205 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.557182    4553 scope.go:122] "RemoveContainer" containerID="83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"
Oct 31 01:14:18.565293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.565273    4553 scope.go:122] "RemoveContainer" containerID="a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"
Oct 31 01:14:18.573094 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.573076    4553 scope.go:122] "RemoveContainer" containerID="3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"
Oct 31 01:14:18.573370 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:14:18.573352    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a\": container with ID starting with 3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a not found: ID does not exist" containerID="3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"
Oct 31 01:14:18.573445 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.573379    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a"} err="failed to get container status \"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a\": rpc error: code = NotFound desc = could not find container \"3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a\": container with ID starting with 3b675228f067dc87032af225fa55e727bc9a5de7eb530a68a2dbc8371778924a not found: ID does not exist"
Oct 31 01:14:18.573445 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.573398    4553 scope.go:122] "RemoveContainer" containerID="83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"
Oct 31 01:14:18.573733 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:14:18.573707    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686\": container with ID starting with 83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686 not found: ID does not exist" containerID="83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"
Oct 31 01:14:18.574293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.573742    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686"} err="failed to get container status \"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686\": rpc error: code = NotFound desc = could not find container \"83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686\": container with ID starting with 83da826cc8b23ee20b9bca9ad8c2a47cb37cb96591ac0492ec833d4204117686 not found: ID does not exist"
Oct 31 01:14:18.574293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.573764    4553 scope.go:122] "RemoveContainer" containerID="a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"
Oct 31 01:14:18.574293 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:14:18.574161    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d\": container with ID starting with a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d not found: ID does not exist" containerID="a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"
Oct 31 01:14:18.574293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.574194    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d"} err="failed to get container status \"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d\": rpc error: code = NotFound desc = could not find container \"a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d\": container with ID starting with a305c1189b8faa6bb8be121b723c85b886abe746cbccad976712460396c6c46d not found: ID does not exist"
Oct 31 01:14:18.575479 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.575461    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:14:18.581632 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:18.581612    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-qr94z"]
Oct 31 01:14:19.199646 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:19.199616    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" path="/var/lib/kubelet/pods/cbe428c6-9f71-4af6-a72b-0b456f1306ed/volumes"
Oct 31 01:14:27.130032 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:14:27.129991    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:14:27.335540 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:27.335511    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:36.772481 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772449    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772753    4553 cpu_manager.go:414] "containerMap: removing container" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="storage-initializer"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772765    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="storage-initializer"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772774    4553 cpu_manager.go:414] "containerMap: removing container" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772778    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772793    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="storage-initializer"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772798    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="storage-initializer"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772806    4553 cpu_manager.go:414] "containerMap: removing container" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kube-rbac-proxy"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772812    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kube-rbac-proxy"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772822    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kube-rbac-proxy"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772826    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kube-rbac-proxy"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772838    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772842    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772888    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kube-rbac-proxy"
Oct 31 01:14:36.772893 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772898    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kube-rbac-proxy"
Oct 31 01:14:36.773468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772905    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe428c6-9f71-4af6-a72b-0b456f1306ed" containerName="kserve-container"
Oct 31 01:14:36.773468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.772912    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="358cc6ca-9a9b-404c-b428-7ee806519e42" containerName="kserve-container"
Oct 31 01:14:36.775895 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.775875    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.776972 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.776944    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:36.782124 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.782103    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 31 01:14:36.782228 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.782153    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:14:36.783114 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.783095    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:14:36.783273 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.783108    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 31 01:14:36.783273 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.783144    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:14:36.783417 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.783160    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:14:36.783417 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.783192    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:14:36.789524 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.789503    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:14:36.832709 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.832664    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.832860 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.832737    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.832860 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.832850    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.832977 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.832883    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.833033 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.832974    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lc6\" (UniqueName: \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934307    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934498 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934351    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934498 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934375    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934498 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934394    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934498 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934412    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lc6\" (UniqueName: \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.934763 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.934746    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.935103 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.935079    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.935180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.935079    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.936901 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.936883    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:36.949047 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:36.949020    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lc6\" (UniqueName: \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\") pod \"isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:37.001497 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:37.001472    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:37.094164 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:37.094087    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:37.371210 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:37.371091    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:14:37.373875 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:14:37.373844    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe93286_c6f8_43de_9fd4_675b09bbf05f.slice/crio-716d9be9b5b61ab67b99b0f6751e90741044d841f1dbe570e8318e8c4f5be015 WatchSource:0}: Error finding container 716d9be9b5b61ab67b99b0f6751e90741044d841f1dbe570e8318e8c4f5be015: Status 404 returned error can't find the container with id 716d9be9b5b61ab67b99b0f6751e90741044d841f1dbe570e8318e8c4f5be015
Oct 31 01:14:37.623441 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:37.623356    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerStarted","Data":"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"}
Oct 31 01:14:37.623441 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:37.623413    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerStarted","Data":"716d9be9b5b61ab67b99b0f6751e90741044d841f1dbe570e8318e8c4f5be015"}
Oct 31 01:14:41.640661 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:41.640627    4553 generic.go:356] "Generic (PLEG): container finished" podID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerID="69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172" exitCode=0
Oct 31 01:14:41.641060 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:41.640708    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerDied","Data":"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"}
Oct 31 01:14:42.647305 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.647270    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerStarted","Data":"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"}
Oct 31 01:14:42.647724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.647332    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerStarted","Data":"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"}
Oct 31 01:14:42.647724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.647637    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:42.647724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.647703    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:42.649279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.649246    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:14:42.676892 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:42.676843    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podStartSLOduration=2.560984959 podStartE2EDuration="6.676830581s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.115845622s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:42.675834391 +0000 UTC m=+2745.584837045" watchObservedRunningTime="2026-10-31 01:14:42.676830581 +0000 UTC m=+2745.585833235"
Oct 31 01:14:43.651541 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:43.651499    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:14:47.538950 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:47.538916    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:48.656537 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:48.656509    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:14:48.657063 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:48.657037    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:14:57.129494 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:14:57.129462    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:14:58.657910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:14:58.657871    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:08.657071 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:08.657034    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:18.657878 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:18.657841    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:27.129586 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:15:27.129549    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:15:28.657970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:28.657931    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:38.657097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:38.657053    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:47.596359 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:47.596321    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:48.657927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:48.657900    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:15:53.672124 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.672094    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:15:53.672507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.672310    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" containerID="cri-o://8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306" gracePeriod=30
Oct 31 01:15:53.672507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.672398    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kube-rbac-proxy" containerID="cri-o://cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688" gracePeriod=30
Oct 31 01:15:53.743045 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.742993    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:53.938609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.938517    4553 generic.go:356] "Generic (PLEG): container finished" podID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerID="cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688" exitCode=0
Oct 31 01:15:53.938609 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:53.938592    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerDied","Data":"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"}
Oct 31 01:15:55.189527 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:55.189484    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Oct 31 01:15:55.194891 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:55.194866    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 31 01:15:57.130371 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:15:57.130326    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:15:57.646800 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.646764    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:15:57.647038 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.647009    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" containerID="cri-o://6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31" gracePeriod=30
Oct 31 01:15:57.647135 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.647094    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kube-rbac-proxy" containerID="cri-o://fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04" gracePeriod=30
Oct 31 01:15:57.715549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.715526    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:57.958070 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.958039    4553 generic.go:356] "Generic (PLEG): container finished" podID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerID="fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04" exitCode=0
Oct 31 01:15:57.958242 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:57.958109    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerDied","Data":"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"}
Oct 31 01:15:58.408838 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.408815    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:15:58.467149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467110    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\") pod \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") "
Oct 31 01:15:58.467327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467156    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\") pod \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") "
Oct 31 01:15:58.467327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467203    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\") pod \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") "
Oct 31 01:15:58.467327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467230    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\" (UniqueName: \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\") pod \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") "
Oct 31 01:15:58.467493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467327    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\") pod \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\" (UID: \"2ea0f89d-b9c1-421b-9512-27ebd06334ef\") "
Oct 31 01:15:58.467546 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467519    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config" pod "2ea0f89d-b9c1-421b-9512-27ebd06334ef" (UID: "2ea0f89d-b9c1-421b-9512-27ebd06334ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:58.467699 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467577    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location" pod "2ea0f89d-b9c1-421b-9512-27ebd06334ef" (UID: "2ea0f89d-b9c1-421b-9512-27ebd06334ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:58.467754 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.467735    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert" pod "2ea0f89d-b9c1-421b-9512-27ebd06334ef" (UID: "2ea0f89d-b9c1-421b-9512-27ebd06334ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:58.469408 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.469389    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls" pod "2ea0f89d-b9c1-421b-9512-27ebd06334ef" (UID: "2ea0f89d-b9c1-421b-9512-27ebd06334ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:58.469515 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.469497    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7" pod "2ea0f89d-b9c1-421b-9512-27ebd06334ef" (UID: "2ea0f89d-b9c1-421b-9512-27ebd06334ef"). InnerVolumeSpecName "kube-api-access-pmdl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:58.568030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.567962    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:58.568030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.567988    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdl7\" (UniqueName: \"kubernetes.io/projected/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-api-access-pmdl7\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:58.568030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.567999    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:58.568030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.568009    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea0f89d-b9c1-421b-9512-27ebd06334ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:58.568030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.568018    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea0f89d-b9c1-421b-9512-27ebd06334ef-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:58.651724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.651672    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Oct 31 01:15:58.657101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.657066    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 31 01:15:58.964756 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.964716    4553 generic.go:356] "Generic (PLEG): container finished" podID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerID="8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306" exitCode=0
Oct 31 01:15:58.964910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.964771    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerDied","Data":"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"}
Oct 31 01:15:58.964910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.964779    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"
Oct 31 01:15:58.964910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.964833    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7" event={"ID":"2ea0f89d-b9c1-421b-9512-27ebd06334ef","Type":"ContainerDied","Data":"51c0c357c8e5dab5ad8ece07fa5d287afeafb31079339a31844a8515240a4a26"}
Oct 31 01:15:58.964910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.964872    4553 scope.go:122] "RemoveContainer" containerID="cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"
Oct 31 01:15:58.973211 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.973105    4553 scope.go:122] "RemoveContainer" containerID="8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"
Oct 31 01:15:58.980670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.980646    4553 scope.go:122] "RemoveContainer" containerID="dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"
Oct 31 01:15:58.987726 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.987703    4553 scope.go:122] "RemoveContainer" containerID="cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"
Oct 31 01:15:58.987995 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:15:58.987976    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688\": container with ID starting with cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688 not found: ID does not exist" containerID="cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"
Oct 31 01:15:58.988058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.988003    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688"} err="failed to get container status \"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688\": rpc error: code = NotFound desc = could not find container \"cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688\": container with ID starting with cc86c44ec741078c1aa60ae272e3d95143c3e9dc6cb7105b5a4574b8d4030688 not found: ID does not exist"
Oct 31 01:15:58.988058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.988021    4553 scope.go:122] "RemoveContainer" containerID="8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"
Oct 31 01:15:58.988314 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:15:58.988297    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306\": container with ID starting with 8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306 not found: ID does not exist" containerID="8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"
Oct 31 01:15:58.988366 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.988321    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306"} err="failed to get container status \"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306\": rpc error: code = NotFound desc = could not find container \"8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306\": container with ID starting with 8996cc7cde58469dfebe759f96467ba1eaa508d89d3ff5d05e304a141a7bf306 not found: ID does not exist"
Oct 31 01:15:58.988366 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.988336    4553 scope.go:122] "RemoveContainer" containerID="dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"
Oct 31 01:15:58.988601 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:15:58.988584    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba\": container with ID starting with dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba not found: ID does not exist" containerID="dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"
Oct 31 01:15:58.988644 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.988609    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba"} err="failed to get container status \"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba\": rpc error: code = NotFound desc = could not find container \"dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba\": container with ID starting with dfada6b429dbed402d10bcfe396222bb902b6a1c9c670d8314ec8e9d0e5040ba not found: ID does not exist"
Oct 31 01:15:58.991463 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:58.991439    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:15:59.001035 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:59.001013    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-899l7"]
Oct 31 01:15:59.199796 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:15:59.199758    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" path="/var/lib/kubelet/pods/2ea0f89d-b9c1-421b-9512-27ebd06334ef/volumes"
Oct 31 01:16:01.775023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.775002    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:16:01.895725 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.895627    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\") pod \"afe93286-c6f8-43de-9fd4-675b09bbf05f\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") "
Oct 31 01:16:01.895725 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.895708    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\") pod \"afe93286-c6f8-43de-9fd4-675b09bbf05f\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") "
Oct 31 01:16:01.895964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.895761    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\") pod \"afe93286-c6f8-43de-9fd4-675b09bbf05f\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") "
Oct 31 01:16:01.895964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.895787    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\" (UniqueName: \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\") pod \"afe93286-c6f8-43de-9fd4-675b09bbf05f\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") "
Oct 31 01:16:01.895964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.895808    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\") pod \"afe93286-c6f8-43de-9fd4-675b09bbf05f\" (UID: \"afe93286-c6f8-43de-9fd4-675b09bbf05f\") "
Oct 31 01:16:01.896135 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.896015    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location" pod "afe93286-c6f8-43de-9fd4-675b09bbf05f" (UID: "afe93286-c6f8-43de-9fd4-675b09bbf05f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:01.896135 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.896120    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config" pod "afe93286-c6f8-43de-9fd4-675b09bbf05f" (UID: "afe93286-c6f8-43de-9fd4-675b09bbf05f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:01.896202 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.896166    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert" pod "afe93286-c6f8-43de-9fd4-675b09bbf05f" (UID: "afe93286-c6f8-43de-9fd4-675b09bbf05f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:01.898053 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.898030    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6" pod "afe93286-c6f8-43de-9fd4-675b09bbf05f" (UID: "afe93286-c6f8-43de-9fd4-675b09bbf05f"). InnerVolumeSpecName "kube-api-access-w8lc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:01.898111 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.898072    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls" pod "afe93286-c6f8-43de-9fd4-675b09bbf05f" (UID: "afe93286-c6f8-43de-9fd4-675b09bbf05f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:01.986528 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.986491    4553 generic.go:356] "Generic (PLEG): container finished" podID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerID="6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31" exitCode=0
Oct 31 01:16:01.986698 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.986566    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"
Oct 31 01:16:01.986698 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.986564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerDied","Data":"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"}
Oct 31 01:16:01.986698 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.986647    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4" event={"ID":"afe93286-c6f8-43de-9fd4-675b09bbf05f","Type":"ContainerDied","Data":"716d9be9b5b61ab67b99b0f6751e90741044d841f1dbe570e8318e8c4f5be015"}
Oct 31 01:16:01.986698 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.986697    4553 scope.go:122] "RemoveContainer" containerID="fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"
Oct 31 01:16:01.995033 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.995017    4553 scope.go:122] "RemoveContainer" containerID="6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"
Oct 31 01:16:01.997244 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.997219    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:01.997326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.997250    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lc6\" (UniqueName: \"kubernetes.io/projected/afe93286-c6f8-43de-9fd4-675b09bbf05f-kube-api-access-w8lc6\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:01.997326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.997266    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe93286-c6f8-43de-9fd4-675b09bbf05f-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:01.997326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.997280    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe93286-c6f8-43de-9fd4-675b09bbf05f-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:01.997326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:01.997295    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe93286-c6f8-43de-9fd4-675b09bbf05f-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:02.002411 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.002394    4553 scope.go:122] "RemoveContainer" containerID="69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"
Oct 31 01:16:02.009778 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.009758    4553 scope.go:122] "RemoveContainer" containerID="fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"
Oct 31 01:16:02.010037 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:16:02.010018    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04\": container with ID starting with fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04 not found: ID does not exist" containerID="fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"
Oct 31 01:16:02.010100 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.010052    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04"} err="failed to get container status \"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04\": rpc error: code = NotFound desc = could not find container \"fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04\": container with ID starting with fe135fa6cc0c0772b44820503ed006ae6afad59f442c85ab337e22a754077e04 not found: ID does not exist"
Oct 31 01:16:02.010100 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.010069    4553 scope.go:122] "RemoveContainer" containerID="6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"
Oct 31 01:16:02.010358 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:16:02.010334    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31\": container with ID starting with 6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31 not found: ID does not exist" containerID="6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"
Oct 31 01:16:02.010420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.010373    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31"} err="failed to get container status \"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31\": rpc error: code = NotFound desc = could not find container \"6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31\": container with ID starting with 6da5250ce43989f34f5ce16dbf0a993b699274a40eda312e4f7a75d8c267bb31 not found: ID does not exist"
Oct 31 01:16:02.010420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.010389    4553 scope.go:122] "RemoveContainer" containerID="69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"
Oct 31 01:16:02.010697 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:16:02.010657    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172\": container with ID starting with 69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172 not found: ID does not exist" containerID="69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"
Oct 31 01:16:02.010827 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.010803    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172"} err="failed to get container status \"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172\": rpc error: code = NotFound desc = could not find container \"69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172\": container with ID starting with 69dce706dbef65d3fc1aba3dbb75bb5d0278782437e1ef5934ab7083cceaf172 not found: ID does not exist"
Oct 31 01:16:02.012375 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.012356    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:16:02.018638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:02.018615    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-predictor-6c8f7d867c-z2xc4"]
Oct 31 01:16:03.199494 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.199459    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" path="/var/lib/kubelet/pods/afe93286-c6f8-43de-9fd4-675b09bbf05f/volumes"
Oct 31 01:16:03.326080 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326046    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:16:03.326341 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326329    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="storage-initializer"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326344    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="storage-initializer"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326353    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326358    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326364    4553 cpu_manager.go:414] "containerMap: removing container" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kube-rbac-proxy"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326369    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kube-rbac-proxy"
Oct 31 01:16:03.326387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326380    4553 cpu_manager.go:414] "containerMap: removing container" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326387    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326405    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kube-rbac-proxy"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326410    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kube-rbac-proxy"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326418    4553 cpu_manager.go:414] "containerMap: removing container" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="storage-initializer"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326423    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="storage-initializer"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326477    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kube-rbac-proxy"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326486    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kserve-container"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326493    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe93286-c6f8-43de-9fd4-675b09bbf05f" containerName="kserve-container"
Oct 31 01:16:03.326571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.326500    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea0f89d-b9c1-421b-9512-27ebd06334ef" containerName="kube-rbac-proxy"
Oct 31 01:16:03.331275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.331256    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.336278 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.336252    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:03.337325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337302    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:16:03.338059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337784    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"storage-config\""
Oct 31 01:16:03.338059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337838    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 31 01:16:03.338059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337879    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:16:03.338059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337894    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 31 01:16:03.338059 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.337942    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:16:03.340845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.340824    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:16:03.406847 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.406823    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.407144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.406867    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.407144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.406891    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbh4\" (UniqueName: \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.407144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.406984    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.478837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.478772    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:03.507763 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.507735    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.507886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.507793    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.507886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.507834    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.507886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.507860    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbh4\" (UniqueName: \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.508226 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.508194    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.508408 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.508390    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.510352 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.510334    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.522274 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.522251    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbh4\" (UniqueName: \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\") pod \"isvc-sklearn-s3-predictor-769b65d644-2hsdd\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.649039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.649012    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:03.925451 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.925420    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:16:03.928394 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:16:03.928362    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c5169b_7f02_4a95_9736_47a715d174db.slice/crio-7d32324aec1b458b0b445d7fed569b4254650a8cd79c12803b2cbb032417c3f3 WatchSource:0}: Error finding container 7d32324aec1b458b0b445d7fed569b4254650a8cd79c12803b2cbb032417c3f3: Status 404 returned error can't find the container with id 7d32324aec1b458b0b445d7fed569b4254650a8cd79c12803b2cbb032417c3f3
Oct 31 01:16:03.997572 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.997543    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerStarted","Data":"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"}
Oct 31 01:16:03.997668 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:03.997596    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerStarted","Data":"7d32324aec1b458b0b445d7fed569b4254650a8cd79c12803b2cbb032417c3f3"}
Oct 31 01:16:04.420919 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.420886    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:16:04.421426 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.421402    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:04.424705 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.424668    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.433965 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.433943    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:16:04.434116 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.433957    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 31 01:16:04.435468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.435446    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:16:04.435566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.435467    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:16:04.435566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.435472    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:16:04.435566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.435446    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 31 01:16:04.435566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.435449    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:16:04.439040 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.439021    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:16:04.468215 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.468186    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:16:04.471622 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.471606    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.484788 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.484766    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:16:04.517282 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.517249    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdxf\" (UniqueName: \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.517444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.517329    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.517444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.517392    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.517526 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.517497    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.517558 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.517543    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.535324 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.535301    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 31 01:16:04.567659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.567622    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:04.618631 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618593    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.618631 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618634    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdxf\" (UniqueName: \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.618827 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618651    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.618911 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618898    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.618949 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618927    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.618983 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.618948    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.619107 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619086    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.619260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619135    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.619260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619185    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.619260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619215    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjnk\" (UniqueName: \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.619555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619530    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.619790 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619762    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.619850 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.619783    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.621745 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.621723    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.633079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.633050    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdxf\" (UniqueName: \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\") pod \"isvc-canary-rollback-v2-predictor-655c46c94b-5qft9\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.719970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.719892    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjnk\" (UniqueName: \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.719970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.719933    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.719970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.719958    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.720239 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.720000    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.720239 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.720036    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.720346 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.720300    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.720685 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.720652    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.720835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.720811    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.722569 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.722540    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.734300 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.734273    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjnk\" (UniqueName: \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\") pod \"isvc-canary-rollback-predictor-59c66949f9-wrngc\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:04.745153 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.745133    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:04.789364 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:04.789330    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:05.024541 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:05.024393    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:16:05.027308 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:16:05.027277    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f30016_c9dd_474b_b593_793bbafa62e4.slice/crio-7855a4afec96a2f47eb610c8635fef5b7b51f65e47416bcc4cd70b515891d55c WatchSource:0}: Error finding container 7855a4afec96a2f47eb610c8635fef5b7b51f65e47416bcc4cd70b515891d55c: Status 404 returned error can't find the container with id 7855a4afec96a2f47eb610c8635fef5b7b51f65e47416bcc4cd70b515891d55c
Oct 31 01:16:05.075022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:05.074999    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:16:05.077366 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:16:05.077342    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17a5903_3c85_4e6f_9c24_28b792a0cd60.slice/crio-4c7707baff209c6d02a9e8c90d3ef89afa6f86915e70d3f62a790c131facd527 WatchSource:0}: Error finding container 4c7707baff209c6d02a9e8c90d3ef89afa6f86915e70d3f62a790c131facd527: Status 404 returned error can't find the container with id 4c7707baff209c6d02a9e8c90d3ef89afa6f86915e70d3f62a790c131facd527
Oct 31 01:16:06.012445 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.012410    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerStarted","Data":"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"}
Oct 31 01:16:06.012952 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.012494    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerStarted","Data":"7855a4afec96a2f47eb610c8635fef5b7b51f65e47416bcc4cd70b515891d55c"}
Oct 31 01:16:06.014001 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.013978    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6c5169b-7f02-4a95-9736-47a715d174db" containerID="d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239" exitCode=0
Oct 31 01:16:06.014133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.014052    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerDied","Data":"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"}
Oct 31 01:16:06.015669 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.015650    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerStarted","Data":"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"}
Oct 31 01:16:06.015753 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:06.015726    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerStarted","Data":"4c7707baff209c6d02a9e8c90d3ef89afa6f86915e70d3f62a790c131facd527"}
Oct 31 01:16:07.021880 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.021846    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerStarted","Data":"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"}
Oct 31 01:16:07.022255 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.021910    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerStarted","Data":"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"}
Oct 31 01:16:07.022431 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.022408    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:07.022502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.022473    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:07.023642 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.023622    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:07.049733 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:07.049692    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podStartSLOduration=2.797023661 podStartE2EDuration="4.049664577s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.252640916s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:07.047265368 +0000 UTC m=+2829.956268021" watchObservedRunningTime="2026-10-31 01:16:07.049664577 +0000 UTC m=+2829.958667269"
Oct 31 01:16:08.025956 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:08.025911    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:09.031345 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:09.031315    4553 generic.go:356] "Generic (PLEG): container finished" podID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerID="2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30" exitCode=0
Oct 31 01:16:09.031668 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:09.031376    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerDied","Data":"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"}
Oct 31 01:16:09.033137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:09.033058    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerID="992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec" exitCode=0
Oct 31 01:16:09.033137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:09.033122    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerDied","Data":"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"}
Oct 31 01:16:10.039631 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.039591    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerStarted","Data":"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"}
Oct 31 01:16:10.040259 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.039662    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerStarted","Data":"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"}
Oct 31 01:16:10.040259 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.039911    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:10.041877 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.041847    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerStarted","Data":"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"}
Oct 31 01:16:10.041948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.041892    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerStarted","Data":"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"}
Oct 31 01:16:10.042087 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.042068    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:10.042133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.042123    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:10.043655 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.043631    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:10.067776 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.067724    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podStartSLOduration=2.215883641 podStartE2EDuration="6.06771144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.851827799s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:10.067258915 +0000 UTC m=+2832.976261564" watchObservedRunningTime="2026-10-31 01:16:10.06771144 +0000 UTC m=+2832.976714093"
Oct 31 01:16:10.094596 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:10.094547    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podStartSLOduration=2.435349736 podStartE2EDuration="6.094535977s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659186241s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:10.093222608 +0000 UTC m=+2833.002225262" watchObservedRunningTime="2026-10-31 01:16:10.094535977 +0000 UTC m=+2833.003538631"
Oct 31 01:16:11.046002 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:11.045952    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:11.046453 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:11.046084    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:11.047425 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:11.047400    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:12.050983 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:12.050933    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:13.030349 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:13.030322    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:16:13.030836 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:13.030808    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:16.052078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:16.052043    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:16:16.052834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:16.052804    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:17.056402 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:17.056373    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:16:17.057099 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:17.057069    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:23.031190 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:23.031147    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:26.053613 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:26.053570    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:27.057080 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:27.057037    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:27.131209 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:16:27.131175    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:16:33.031786 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:33.031739    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:36.053065 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:36.053013    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:37.057992 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:37.057949    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:43.031222 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:43.031177    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:46.053571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:46.053529    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:47.057052 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:47.057014    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:47.644363 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:47.644313    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:53.031553 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:53.031514    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:16:56.053477 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:56.053439    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:16:57.057432 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:16:57.057386    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:16:57.130508 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:16:57.130473    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:17:03.031639 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:03.031593    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:17:06.052933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:06.052882    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:17:07.057838 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:07.057794    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:17:13.031567 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:13.031523    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 31 01:17:16.053067 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:16.053018    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:17:17.057595 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:17.057549    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 31 01:17:23.031833 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:23.031800    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:17:24.267165 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:24.267134    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:17:24.267549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:24.267354    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container" containerID="cri-o://b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6" gracePeriod=30
Oct 31 01:17:24.267549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:24.267380    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kube-rbac-proxy" containerID="cri-o://1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a" gracePeriod=30
Oct 31 01:17:24.284627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:24.284604    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:24.426383 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:24.426349    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:25.359754 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:25.359719    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6c5169b-7f02-4a95-9736-47a715d174db" containerID="1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a" exitCode=0
Oct 31 01:17:25.360150 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:25.359795    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerDied","Data":"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"}
Oct 31 01:17:26.053366 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:26.053328    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:17:26.271384 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:26.271353    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:17:26.271664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:26.271629    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container" containerID="cri-o://7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45" gracePeriod=30
Oct 31 01:17:26.271845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:26.271646    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kube-rbac-proxy" containerID="cri-o://e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188" gracePeriod=30
Oct 31 01:17:26.272746 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:26.272728    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:17:27.051619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:27.051580    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Oct 31 01:17:27.129263 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:27.129227    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:17:27.370293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:27.370210    4553 generic.go:356] "Generic (PLEG): container finished" podID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerID="e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188" exitCode=0
Oct 31 01:17:27.370293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:27.370267    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerDied","Data":"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"}
Oct 31 01:17:28.026561 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:28.026519    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Oct 31 01:17:29.205457 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.205433    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:17:29.315318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315221    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\") pod \"d6c5169b-7f02-4a95-9736-47a715d174db\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") "
Oct 31 01:17:29.315318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315316    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\") pod \"d6c5169b-7f02-4a95-9736-47a715d174db\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") "
Oct 31 01:17:29.315513 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315339    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\") pod \"d6c5169b-7f02-4a95-9736-47a715d174db\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") "
Oct 31 01:17:29.315513 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315374    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\" (UniqueName: \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\") pod \"d6c5169b-7f02-4a95-9736-47a715d174db\" (UID: \"d6c5169b-7f02-4a95-9736-47a715d174db\") "
Oct 31 01:17:29.315740 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315704    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location" pod "d6c5169b-7f02-4a95-9736-47a715d174db" (UID: "d6c5169b-7f02-4a95-9736-47a715d174db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:29.315740 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.315733    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config" pod "d6c5169b-7f02-4a95-9736-47a715d174db" (UID: "d6c5169b-7f02-4a95-9736-47a715d174db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:29.317532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.317509    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls" pod "d6c5169b-7f02-4a95-9736-47a715d174db" (UID: "d6c5169b-7f02-4a95-9736-47a715d174db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:29.317630 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.317606    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4" pod "d6c5169b-7f02-4a95-9736-47a715d174db" (UID: "d6c5169b-7f02-4a95-9736-47a715d174db"). InnerVolumeSpecName "kube-api-access-jhbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:29.380337 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.380299    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6c5169b-7f02-4a95-9736-47a715d174db" containerID="b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6" exitCode=0
Oct 31 01:17:29.380503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.380366    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"
Oct 31 01:17:29.380503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.380381    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerDied","Data":"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"}
Oct 31 01:17:29.380503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.380440    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd" event={"ID":"d6c5169b-7f02-4a95-9736-47a715d174db","Type":"ContainerDied","Data":"7d32324aec1b458b0b445d7fed569b4254650a8cd79c12803b2cbb032417c3f3"}
Oct 31 01:17:29.380503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.380465    4553 scope.go:122] "RemoveContainer" containerID="1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"
Oct 31 01:17:29.392018 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.391993    4553 scope.go:122] "RemoveContainer" containerID="b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"
Oct 31 01:17:29.399785 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.399742    4553 scope.go:122] "RemoveContainer" containerID="d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"
Oct 31 01:17:29.407039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407023    4553 scope.go:122] "RemoveContainer" containerID="1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"
Oct 31 01:17:29.407319 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:29.407301    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a\": container with ID starting with 1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a not found: ID does not exist" containerID="1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"
Oct 31 01:17:29.407362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407331    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a"} err="failed to get container status \"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a\": rpc error: code = NotFound desc = could not find container \"1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a\": container with ID starting with 1e5e44a707b0e1885d8c9e7c15951842edffdb9bc98cecb8d151751e75efc03a not found: ID does not exist"
Oct 31 01:17:29.407362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407348    4553 scope.go:122] "RemoveContainer" containerID="b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"
Oct 31 01:17:29.407591 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:29.407570    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6\": container with ID starting with b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6 not found: ID does not exist" containerID="b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"
Oct 31 01:17:29.407636 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407597    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6"} err="failed to get container status \"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6\": rpc error: code = NotFound desc = could not find container \"b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6\": container with ID starting with b08d2fc99ccd0c9f26e59ce8ccbb3c2ce72b29dff982417dfb4fe6efd3602fc6 not found: ID does not exist"
Oct 31 01:17:29.407636 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407616    4553 scope.go:122] "RemoveContainer" containerID="d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"
Oct 31 01:17:29.407900 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:29.407882    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239\": container with ID starting with d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239 not found: ID does not exist" containerID="d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"
Oct 31 01:17:29.407966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.407908    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239"} err="failed to get container status \"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239\": rpc error: code = NotFound desc = could not find container \"d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239\": container with ID starting with d994d897c6fcb8e16c5e932c0c35751d9de8a0ff45d3a31386ce84bf33d1a239 not found: ID does not exist"
Oct 31 01:17:29.412349 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.412329    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:17:29.416363 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.416347    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c5169b-7f02-4a95-9736-47a715d174db-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:29.416438 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.416365    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c5169b-7f02-4a95-9736-47a715d174db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:29.416438 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.416376    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbh4\" (UniqueName: \"kubernetes.io/projected/d6c5169b-7f02-4a95-9736-47a715d174db-kube-api-access-jhbh4\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:29.416438 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.416385    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c5169b-7f02-4a95-9736-47a715d174db-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:29.419012 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:29.418994    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-predictor-769b65d644-2hsdd"]
Oct 31 01:17:31.200420 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.200388    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" path="/var/lib/kubelet/pods/d6c5169b-7f02-4a95-9736-47a715d174db/volumes"
Oct 31 01:17:31.211727 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.211707    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:17:31.231238 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231207    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\") pod \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") "
Oct 31 01:17:31.231371 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231250    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\") pod \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") "
Oct 31 01:17:31.231371 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231275    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\" (UniqueName: \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\") pod \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") "
Oct 31 01:17:31.231371 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231298    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\") pod \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") "
Oct 31 01:17:31.231538 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231455    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\") pod \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\" (UID: \"b17a5903-3c85-4e6f-9c24-28b792a0cd60\") "
Oct 31 01:17:31.231590 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231545    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config" pod "b17a5903-3c85-4e6f-9c24-28b792a0cd60" (UID: "b17a5903-3c85-4e6f-9c24-28b792a0cd60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:31.231666 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231646    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location" pod "b17a5903-3c85-4e6f-9c24-28b792a0cd60" (UID: "b17a5903-3c85-4e6f-9c24-28b792a0cd60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:31.231836 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231810    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert" pod "b17a5903-3c85-4e6f-9c24-28b792a0cd60" (UID: "b17a5903-3c85-4e6f-9c24-28b792a0cd60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:31.231906 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231842    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:31.231906 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.231863    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:31.234155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.234120    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls" pod "b17a5903-3c85-4e6f-9c24-28b792a0cd60" (UID: "b17a5903-3c85-4e6f-9c24-28b792a0cd60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:31.234155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.234139    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk" pod "b17a5903-3c85-4e6f-9c24-28b792a0cd60" (UID: "b17a5903-3c85-4e6f-9c24-28b792a0cd60"). InnerVolumeSpecName "kube-api-access-xqjnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:31.332347 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.332262    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17a5903-3c85-4e6f-9c24-28b792a0cd60-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:31.332347 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.332293    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b17a5903-3c85-4e6f-9c24-28b792a0cd60-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:31.332347 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.332303    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjnk\" (UniqueName: \"kubernetes.io/projected/b17a5903-3c85-4e6f-9c24-28b792a0cd60-kube-api-access-xqjnk\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:31.391580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.391542    4553 generic.go:356] "Generic (PLEG): container finished" podID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerID="7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45" exitCode=0
Oct 31 01:17:31.391766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.391595    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"
Oct 31 01:17:31.391766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.391617    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerDied","Data":"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"}
Oct 31 01:17:31.391766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.391716    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc" event={"ID":"b17a5903-3c85-4e6f-9c24-28b792a0cd60","Type":"ContainerDied","Data":"4c7707baff209c6d02a9e8c90d3ef89afa6f86915e70d3f62a790c131facd527"}
Oct 31 01:17:31.391766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.391745    4553 scope.go:122] "RemoveContainer" containerID="e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"
Oct 31 01:17:31.400204 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.400188    4553 scope.go:122] "RemoveContainer" containerID="7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"
Oct 31 01:17:31.407959 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.407941    4553 scope.go:122] "RemoveContainer" containerID="2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"
Oct 31 01:17:31.415117 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415096    4553 scope.go:122] "RemoveContainer" containerID="e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"
Oct 31 01:17:31.415366 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:31.415340    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188\": container with ID starting with e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188 not found: ID does not exist" containerID="e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"
Oct 31 01:17:31.415419 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415369    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188"} err="failed to get container status \"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188\": rpc error: code = NotFound desc = could not find container \"e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188\": container with ID starting with e337c3043c65aa90aa5f47686783129d5616299f78b54b9833204c494fdb8188 not found: ID does not exist"
Oct 31 01:17:31.415419 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415388    4553 scope.go:122] "RemoveContainer" containerID="7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"
Oct 31 01:17:31.415656 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:31.415633    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45\": container with ID starting with 7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45 not found: ID does not exist" containerID="7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"
Oct 31 01:17:31.415747 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415660    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45"} err="failed to get container status \"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45\": rpc error: code = NotFound desc = could not find container \"7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45\": container with ID starting with 7ad6c45dab2b03f644d05ab460bca1b041a4f306aad3d5cf1b1ec48950cc9c45 not found: ID does not exist"
Oct 31 01:17:31.415747 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415706    4553 scope.go:122] "RemoveContainer" containerID="2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"
Oct 31 01:17:31.415975 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:31.415958    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30\": container with ID starting with 2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30 not found: ID does not exist" containerID="2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"
Oct 31 01:17:31.416020 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.415981    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30"} err="failed to get container status \"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30\": rpc error: code = NotFound desc = could not find container \"2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30\": container with ID starting with 2983c573127cca146b77c41a30fb68b36500753ffb62c5b674a8eb79cddd5e30 not found: ID does not exist"
Oct 31 01:17:31.418647 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.418626    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:17:31.424587 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.424568    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-wrngc"]
Oct 31 01:17:31.891560 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891522    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:17:31.891923 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891904    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kube-rbac-proxy"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891925    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kube-rbac-proxy"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891942    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="storage-initializer"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891950    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="storage-initializer"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891975    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891983    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.891996    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892004    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container"
Oct 31 01:17:31.892023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892021    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="storage-initializer"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892029    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="storage-initializer"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892039    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kube-rbac-proxy"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892045    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kube-rbac-proxy"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892127    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kserve-container"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892138    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kserve-container"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892149    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c5169b-7f02-4a95-9736-47a715d174db" containerName="kube-rbac-proxy"
Oct 31 01:17:31.892415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.892160    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" containerName="kube-rbac-proxy"
Oct 31 01:17:31.896816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.896796    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:31.907551 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.907524    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 31 01:17:31.907670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.907614    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"storage-config\""
Oct 31 01:17:31.907670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.907665    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 31 01:17:31.908557 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.908531    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:17:31.908647 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.908637    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:17:31.908766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.908745    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:17:31.908871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.908782    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:17:31.908871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.908787    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:17:31.913361 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.913333    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:31.936605 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.936575    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:31.936761 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.936644    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:31.936761 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.936721    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:31.936761 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.936747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwc5\" (UniqueName: \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:31.936910 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:31.936840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038000    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038039    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038086    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038113    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwc5\" (UniqueName: \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038138    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038433 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:32.038267    4553 secret.go:189] Couldn't get secret e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Oct 31 01:17:32.038433 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:32.038375    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls podName:a46aa7a2-9056-4c2a-8084-056f59da3083 nodeName:}" failed. No retries permitted until 2026-10-31 01:17:32.53835401 +0000 UTC m=+2915.447356659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Oct 31 01:17:32.038655 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038633    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038812 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038794    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.038852 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.038829    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.054441 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.054411    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwc5\" (UniqueName: \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.194361 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.194330    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:32.541736 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.541626    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.544277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.544253    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:32.814755 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:32.814661    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:33.095038 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:33.095016    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:17:33.096975 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:17:33.096949    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46aa7a2_9056_4c2a_8084_056f59da3083.slice/crio-1c3474cb221906b277f402c44cab3edbd3b9b8e75d768de826d7c7630e73d437 WatchSource:0}: Error finding container 1c3474cb221906b277f402c44cab3edbd3b9b8e75d768de826d7c7630e73d437: Status 404 returned error can't find the container with id 1c3474cb221906b277f402c44cab3edbd3b9b8e75d768de826d7c7630e73d437
Oct 31 01:17:33.200304 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:33.200271    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17a5903-3c85-4e6f-9c24-28b792a0cd60" path="/var/lib/kubelet/pods/b17a5903-3c85-4e6f-9c24-28b792a0cd60/volumes"
Oct 31 01:17:33.402031 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:33.401992    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerStarted","Data":"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"}
Oct 31 01:17:33.402223 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:33.402077    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerStarted","Data":"1c3474cb221906b277f402c44cab3edbd3b9b8e75d768de826d7c7630e73d437"}
Oct 31 01:17:34.407517 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.407483    4553 generic.go:356] "Generic (PLEG): container finished" podID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerID="40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47" exitCode=0
Oct 31 01:17:34.407929 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.407524    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerDied","Data":"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"}
Oct 31 01:17:34.554977 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.554950    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:17:34.555232 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.555205    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" containerID="cri-o://b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf" gracePeriod=30
Oct 31 01:17:34.555362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.555229    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kube-rbac-proxy" containerID="cri-o://0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261" gracePeriod=30
Oct 31 01:17:34.607292 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.607263    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:17:34.610623 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.610600    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.616524 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.616502    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 31 01:17:34.628666 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.628643    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:17:34.653217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.653192    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:34.657424 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.657403    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.657498 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.657442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.657559 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.657526    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.657605 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.657592    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk6x\" (UniqueName: \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.657647 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.657613    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758284 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758213    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk6x\" (UniqueName: \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758284 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758247    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758284 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758271    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758535 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758308    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758535 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758361    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.758731 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.758697    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.759048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.759027    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.759118 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.759027    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.760911 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.760893    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.773235 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.773212    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk6x\" (UniqueName: \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\") pod \"isvc-canary-rollback-predictor-59c66949f9-pvcfr\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:34.927839 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:34.927807    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:35.210757 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.210732    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:17:35.212629 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:17:35.212599    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9813b6e9_a988_4440_baa5_30d3bbd52587.slice/crio-efb7c304c688aa9493dd13325678691a37500a7a92be47287ac52b9a39262f16 WatchSource:0}: Error finding container efb7c304c688aa9493dd13325678691a37500a7a92be47287ac52b9a39262f16: Status 404 returned error can't find the container with id efb7c304c688aa9493dd13325678691a37500a7a92be47287ac52b9a39262f16
Oct 31 01:17:35.415292 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.415254    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerStarted","Data":"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"}
Oct 31 01:17:35.415814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.415331    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerStarted","Data":"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"}
Oct 31 01:17:35.415814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.415431    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:35.415814 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.415491    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:35.416855 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.416818    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerStarted","Data":"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"}
Oct 31 01:17:35.416983 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.416877    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerStarted","Data":"efb7c304c688aa9493dd13325678691a37500a7a92be47287ac52b9a39262f16"}
Oct 31 01:17:35.417307 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.417283    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:17:35.419170 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.419150    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerID="0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261" exitCode=0
Oct 31 01:17:35.419258 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.419181    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerDied","Data":"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"}
Oct 31 01:17:35.442332 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:35.442292    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podStartSLOduration=3.427949661 podStartE2EDuration="4.442281971s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.01433231s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:35.441953879 +0000 UTC m=+2918.350956546" watchObservedRunningTime="2026-10-31 01:17:35.442281971 +0000 UTC m=+2918.351284625"
Oct 31 01:17:36.046888 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:36.046846    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Oct 31 01:17:36.053037 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:36.052992    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 31 01:17:36.424144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:36.424097    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:17:39.207846 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.207818    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:17:39.294031 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.293996    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\") pod \"d6f30016-c9dd-474b-b593-793bbafa62e4\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") "
Oct 31 01:17:39.294180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294041    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\" (UniqueName: \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\") pod \"d6f30016-c9dd-474b-b593-793bbafa62e4\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") "
Oct 31 01:17:39.294180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294061    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\") pod \"d6f30016-c9dd-474b-b593-793bbafa62e4\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") "
Oct 31 01:17:39.294180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294109    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\") pod \"d6f30016-c9dd-474b-b593-793bbafa62e4\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") "
Oct 31 01:17:39.294180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294161    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\") pod \"d6f30016-c9dd-474b-b593-793bbafa62e4\" (UID: \"d6f30016-c9dd-474b-b593-793bbafa62e4\") "
Oct 31 01:17:39.294497 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294458    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert" pod "d6f30016-c9dd-474b-b593-793bbafa62e4" (UID: "d6f30016-c9dd-474b-b593-793bbafa62e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:39.294606 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294469    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config" pod "d6f30016-c9dd-474b-b593-793bbafa62e4" (UID: "d6f30016-c9dd-474b-b593-793bbafa62e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:39.294606 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.294478    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location" pod "d6f30016-c9dd-474b-b593-793bbafa62e4" (UID: "d6f30016-c9dd-474b-b593-793bbafa62e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:39.296292 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.296270    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf" pod "d6f30016-c9dd-474b-b593-793bbafa62e4" (UID: "d6f30016-c9dd-474b-b593-793bbafa62e4"). InnerVolumeSpecName "kube-api-access-nwdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:39.296356 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.296290    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls" pod "d6f30016-c9dd-474b-b593-793bbafa62e4" (UID: "d6f30016-c9dd-474b-b593-793bbafa62e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:39.394834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.394764    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f30016-c9dd-474b-b593-793bbafa62e4-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:39.394834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.394797    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:39.394834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.394809    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdxf\" (UniqueName: \"kubernetes.io/projected/d6f30016-c9dd-474b-b593-793bbafa62e4-kube-api-access-nwdxf\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:39.394834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.394822    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f30016-c9dd-474b-b593-793bbafa62e4-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:39.394834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.394835    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f30016-c9dd-474b-b593-793bbafa62e4-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:39.438076 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.438046    4553 generic.go:356] "Generic (PLEG): container finished" podID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerID="2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756" exitCode=0
Oct 31 01:17:39.438186 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.438123    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerDied","Data":"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"}
Oct 31 01:17:39.440193 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.440166    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerID="b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf" exitCode=0
Oct 31 01:17:39.440296 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.440236    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"
Oct 31 01:17:39.440296 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.440247    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerDied","Data":"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"}
Oct 31 01:17:39.440415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.440302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9" event={"ID":"d6f30016-c9dd-474b-b593-793bbafa62e4","Type":"ContainerDied","Data":"7855a4afec96a2f47eb610c8635fef5b7b51f65e47416bcc4cd70b515891d55c"}
Oct 31 01:17:39.440415 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.440327    4553 scope.go:122] "RemoveContainer" containerID="0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"
Oct 31 01:17:39.449669 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.449652    4553 scope.go:122] "RemoveContainer" containerID="b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"
Oct 31 01:17:39.461084 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.461055    4553 scope.go:122] "RemoveContainer" containerID="992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"
Oct 31 01:17:39.470748 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.470729    4553 scope.go:122] "RemoveContainer" containerID="0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"
Oct 31 01:17:39.471026 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:39.471005    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261\": container with ID starting with 0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261 not found: ID does not exist" containerID="0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"
Oct 31 01:17:39.471092 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.471039    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261"} err="failed to get container status \"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261\": rpc error: code = NotFound desc = could not find container \"0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261\": container with ID starting with 0529019c2138f66a246dfc1bcac25db8195edc87cc0bd1f3d75dec7b8d9f3261 not found: ID does not exist"
Oct 31 01:17:39.471092 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.471063    4553 scope.go:122] "RemoveContainer" containerID="b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"
Oct 31 01:17:39.471310 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:39.471294    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf\": container with ID starting with b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf not found: ID does not exist" containerID="b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"
Oct 31 01:17:39.471378 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.471313    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf"} err="failed to get container status \"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf\": rpc error: code = NotFound desc = could not find container \"b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf\": container with ID starting with b69ab0c65438d546cc36f9a7792cb775961db225dbddc4cd835d7d1689c71faf not found: ID does not exist"
Oct 31 01:17:39.471378 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.471326    4553 scope.go:122] "RemoveContainer" containerID="992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"
Oct 31 01:17:39.471546 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:39.471525    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec\": container with ID starting with 992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec not found: ID does not exist" containerID="992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"
Oct 31 01:17:39.471601 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.471556    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec"} err="failed to get container status \"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec\": rpc error: code = NotFound desc = could not find container \"992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec\": container with ID starting with 992028b4e47c63343127ad845578babd7cb5d9e15c455bf4b2e1d2df7c6bbaec not found: ID does not exist"
Oct 31 01:17:39.481533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.481509    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:17:39.489921 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:39.489898    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-v2-predictor-655c46c94b-5qft9"]
Oct 31 01:17:40.446856 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:40.446817    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerStarted","Data":"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"}
Oct 31 01:17:40.447351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:40.446901    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerStarted","Data":"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"}
Oct 31 01:17:40.447351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:40.447127    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:40.477167 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:40.477120    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podStartSLOduration=2.540456195 podStartE2EDuration="6.47710805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936651855s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:40.475257252 +0000 UTC m=+2923.384259906" watchObservedRunningTime="2026-10-31 01:17:40.47710805 +0000 UTC m=+2923.386110704"
Oct 31 01:17:41.199473 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:41.199438    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" path="/var/lib/kubelet/pods/d6f30016-c9dd-474b-b593-793bbafa62e4/volumes"
Oct 31 01:17:41.428313 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:41.428285    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:17:41.428729 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:41.428704    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:17:41.453405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:41.453321    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:41.454670 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:41.454644    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:17:42.457050 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:42.457011    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:17:47.461756 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:47.461722    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:17:47.462225 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:47.462165    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:17:47.720875 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:47.720796    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:51.429003 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:51.428964    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:17:57.129482 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:17:57.129436    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:17:57.462357 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:17:57.462317    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:01.429236 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:01.429200    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:07.462199 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:07.462160    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:11.428930 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:11.428887    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:17.462362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:17.462324    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:21.428745 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:21.428707    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:27.130448 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:18:27.130409    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:18:27.463071 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:27.463037    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:28.035444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:28.035411    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:31.429690 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:31.429646    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:37.462293 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:37.462256    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:41.428831 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:41.428789    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:47.463151 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:47.463114    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:47.772369 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:47.772262    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:48.195653 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:48.195610    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 31 01:18:51.195538 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:51.195486    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Oct 31 01:18:57.130307 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:18:57.130261    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:18:57.425577 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:57.425547    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:18:57.430447 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:57.430425    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:18:58.196664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:18:58.196630    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:19:01.198822 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:01.198792    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:19:02.053639 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.053606    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:19:02.053889 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.053863    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container" containerID="cri-o://0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d" gracePeriod=30
Oct 31 01:19:02.053962 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.053928    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kube-rbac-proxy" containerID="cri-o://6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615" gracePeriod=30
Oct 31 01:19:02.183776 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.183734    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:02.783749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.783715    4553 generic.go:356] "Generic (PLEG): container finished" podID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerID="6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615" exitCode=0
Oct 31 01:19:02.784156 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:02.783781    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerDied","Data":"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"}
Oct 31 01:19:03.311150 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.311106    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:03.312282 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312261    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:03.312562 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312550    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container"
Oct 31 01:19:03.312619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312563    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container"
Oct 31 01:19:03.312619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312588    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="storage-initializer"
Oct 31 01:19:03.312619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312593    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="storage-initializer"
Oct 31 01:19:03.312619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312601    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kube-rbac-proxy"
Oct 31 01:19:03.312619 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312606    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kube-rbac-proxy"
Oct 31 01:19:03.312876 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312667    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kserve-container"
Oct 31 01:19:03.312876 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.312695    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f30016-c9dd-474b-b593-793bbafa62e4" containerName="kube-rbac-proxy"
Oct 31 01:19:03.315734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.315716    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.322804 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.322655    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Oct 31 01:19:03.323258 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.323238    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Oct 31 01:19:03.333000 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.332975    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:03.350296 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.350272    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.350425 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.350347    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.350487 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.350435    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.350558 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.350525    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgstw\" (UniqueName: \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.451503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.451473    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.451503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.451505    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.451796 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.451528    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgstw\" (UniqueName: \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.451796 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.451572    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.452023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.451999    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.452275 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.452254    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.454212 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.454193    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.459459 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.459429    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:03.466450 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.466427    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgstw\" (UniqueName: \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.633711 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.633613    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:03.912049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:03.912021    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:03.914076 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:19:03.914052    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18553d82_1ce7_45b0_a76c_c5c9755bbd76.slice/crio-311bd5c025d3ad7c39ff50bb3c3bca8d65705601c43ec117ddb5ba03fa0b9527 WatchSource:0}: Error finding container 311bd5c025d3ad7c39ff50bb3c3bca8d65705601c43ec117ddb5ba03fa0b9527: Status 404 returned error can't find the container with id 311bd5c025d3ad7c39ff50bb3c3bca8d65705601c43ec117ddb5ba03fa0b9527
Oct 31 01:19:04.794198 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.794157    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerStarted","Data":"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"}
Oct 31 01:19:04.794370 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.794227    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerStarted","Data":"311bd5c025d3ad7c39ff50bb3c3bca8d65705601c43ec117ddb5ba03fa0b9527"}
Oct 31 01:19:04.885041 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.885002    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:19:04.885267 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.885241    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container" containerID="cri-o://3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b" gracePeriod=30
Oct 31 01:19:04.885357 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.885276    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kube-rbac-proxy" containerID="cri-o://1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50" gracePeriod=30
Oct 31 01:19:04.960128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:04.960095    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:05.800580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:05.800541    4553 generic.go:356] "Generic (PLEG): container finished" podID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerID="1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50" exitCode=0
Oct 31 01:19:05.800784 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:05.800620    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerDied","Data":"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"}
Oct 31 01:19:06.424409 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:06.424365    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Oct 31 01:19:07.096915 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.096894    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:19:07.184948 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.184915    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\") pod \"a46aa7a2-9056-4c2a-8084-056f59da3083\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") "
Oct 31 01:19:07.185140 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.184979    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\" (UniqueName: \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\") pod \"a46aa7a2-9056-4c2a-8084-056f59da3083\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") "
Oct 31 01:19:07.185140 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185014    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\") pod \"a46aa7a2-9056-4c2a-8084-056f59da3083\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") "
Oct 31 01:19:07.185140 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185058    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\") pod \"a46aa7a2-9056-4c2a-8084-056f59da3083\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") "
Oct 31 01:19:07.185140 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185094    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") pod \"a46aa7a2-9056-4c2a-8084-056f59da3083\" (UID: \"a46aa7a2-9056-4c2a-8084-056f59da3083\") "
Oct 31 01:19:07.185364 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185339    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config" pod "a46aa7a2-9056-4c2a-8084-056f59da3083" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:07.185423 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185390    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert" pod "a46aa7a2-9056-4c2a-8084-056f59da3083" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:07.185423 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.185401    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location" pod "a46aa7a2-9056-4c2a-8084-056f59da3083" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:07.187266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.187244    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls" pod "a46aa7a2-9056-4c2a-8084-056f59da3083" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:07.187374 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.187356    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5" pod "a46aa7a2-9056-4c2a-8084-056f59da3083" (UID: "a46aa7a2-9056-4c2a-8084-056f59da3083"). InnerVolumeSpecName "kube-api-access-lwwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:07.286526 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.286487    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:07.286526 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.286521    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwc5\" (UniqueName: \"kubernetes.io/projected/a46aa7a2-9056-4c2a-8084-056f59da3083-kube-api-access-lwwc5\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:07.286526 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.286534    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46aa7a2-9056-4c2a-8084-056f59da3083-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:07.286794 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.286547    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a46aa7a2-9056-4c2a-8084-056f59da3083-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:07.286794 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.286560    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a46aa7a2-9056-4c2a-8084-056f59da3083-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:07.457734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.457625    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Oct 31 01:19:07.808816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.808743    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/0.log"
Oct 31 01:19:07.808816 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.808782    4553 generic.go:356] "Generic (PLEG): container finished" podID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerID="38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d" exitCode=1
Oct 31 01:19:07.809013 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.808866    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerDied","Data":"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"}
Oct 31 01:19:07.810994 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.810969    4553 generic.go:356] "Generic (PLEG): container finished" podID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerID="0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d" exitCode=0
Oct 31 01:19:07.811102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.811016    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"
Oct 31 01:19:07.811102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.811039    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerDied","Data":"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"}
Oct 31 01:19:07.811102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.811088    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w" event={"ID":"a46aa7a2-9056-4c2a-8084-056f59da3083","Type":"ContainerDied","Data":"1c3474cb221906b277f402c44cab3edbd3b9b8e75d768de826d7c7630e73d437"}
Oct 31 01:19:07.811264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.811117    4553 scope.go:122] "RemoveContainer" containerID="6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"
Oct 31 01:19:07.819317 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.819295    4553 scope.go:122] "RemoveContainer" containerID="0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"
Oct 31 01:19:07.827242 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.827223    4553 scope.go:122] "RemoveContainer" containerID="40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"
Oct 31 01:19:07.835785 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.835763    4553 scope.go:122] "RemoveContainer" containerID="6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"
Oct 31 01:19:07.836080 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:07.836060    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615\": container with ID starting with 6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615 not found: ID does not exist" containerID="6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"
Oct 31 01:19:07.836186 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.836090    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615"} err="failed to get container status \"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615\": rpc error: code = NotFound desc = could not find container \"6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615\": container with ID starting with 6812a6ab5f2eab5a74f0a039db62fd182fdec126b9f68c784eaf924461ef0615 not found: ID does not exist"
Oct 31 01:19:07.836186 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.836112    4553 scope.go:122] "RemoveContainer" containerID="0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"
Oct 31 01:19:07.836864 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:07.836836    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d\": container with ID starting with 0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d not found: ID does not exist" containerID="0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"
Oct 31 01:19:07.836973 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.836870    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d"} err="failed to get container status \"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d\": rpc error: code = NotFound desc = could not find container \"0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d\": container with ID starting with 0d3aa126a71e59c0c33a28b15013843b2421045c00238cb1e37b0291e8e93c3d not found: ID does not exist"
Oct 31 01:19:07.836973 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.836891    4553 scope.go:122] "RemoveContainer" containerID="40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"
Oct 31 01:19:07.837172 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:07.837147    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47\": container with ID starting with 40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47 not found: ID does not exist" containerID="40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"
Oct 31 01:19:07.837261 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.837179    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47"} err="failed to get container status \"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47\": rpc error: code = NotFound desc = could not find container \"40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47\": container with ID starting with 40e8e5330fed751ea131e976d4f34ecb29197d60cc4b2697858d48a348793e47 not found: ID does not exist"
Oct 31 01:19:07.854461 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.854435    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:19:07.861830 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:07.861809    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-pass-predictor-56594b59d6-8zz6w"]
Oct 31 01:19:08.816945 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:08.816869    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/0.log"
Oct 31 01:19:08.817366 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:08.816960    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerStarted","Data":"62c5c586f30875d2982ab973a09cf89d9368e9ea4b182a0aaaff39b91a86d379"}
Oct 31 01:19:09.200280 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:09.200246    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" path="/var/lib/kubelet/pods/a46aa7a2-9056-4c2a-8084-056f59da3083/volumes"
Oct 31 01:19:10.027668 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.027644    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:19:10.111058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.110971    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\" (UniqueName: \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\") pod \"9813b6e9-a988-4440-baa5-30d3bbd52587\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") "
Oct 31 01:19:10.111058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111013    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\") pod \"9813b6e9-a988-4440-baa5-30d3bbd52587\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") "
Oct 31 01:19:10.111058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111043    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\") pod \"9813b6e9-a988-4440-baa5-30d3bbd52587\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") "
Oct 31 01:19:10.111361 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111093    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\") pod \"9813b6e9-a988-4440-baa5-30d3bbd52587\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") "
Oct 31 01:19:10.111361 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111218    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\") pod \"9813b6e9-a988-4440-baa5-30d3bbd52587\" (UID: \"9813b6e9-a988-4440-baa5-30d3bbd52587\") "
Oct 31 01:19:10.111468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111368    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location" pod "9813b6e9-a988-4440-baa5-30d3bbd52587" (UID: "9813b6e9-a988-4440-baa5-30d3bbd52587"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:10.111519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111460    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config" pod "9813b6e9-a988-4440-baa5-30d3bbd52587" (UID: "9813b6e9-a988-4440-baa5-30d3bbd52587"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:10.111580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111556    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert" pod "9813b6e9-a988-4440-baa5-30d3bbd52587" (UID: "9813b6e9-a988-4440-baa5-30d3bbd52587"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:10.111658 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111641    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9813b6e9-a988-4440-baa5-30d3bbd52587-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:10.111736 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111661    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:10.111736 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.111697    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9813b6e9-a988-4440-baa5-30d3bbd52587-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:10.113937 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.113912    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls" pod "9813b6e9-a988-4440-baa5-30d3bbd52587" (UID: "9813b6e9-a988-4440-baa5-30d3bbd52587"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:10.114042 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.113917    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x" pod "9813b6e9-a988-4440-baa5-30d3bbd52587" (UID: "9813b6e9-a988-4440-baa5-30d3bbd52587"). InnerVolumeSpecName "kube-api-access-8tk6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:10.212925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.212863    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9813b6e9-a988-4440-baa5-30d3bbd52587-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:10.212925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.212917    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tk6x\" (UniqueName: \"kubernetes.io/projected/9813b6e9-a988-4440-baa5-30d3bbd52587-kube-api-access-8tk6x\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:10.827749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.827617    4553 generic.go:356] "Generic (PLEG): container finished" podID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerID="3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b" exitCode=0
Oct 31 01:19:10.827749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.827691    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"
Oct 31 01:19:10.827749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.827713    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerDied","Data":"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"}
Oct 31 01:19:10.828008 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.827789    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr" event={"ID":"9813b6e9-a988-4440-baa5-30d3bbd52587","Type":"ContainerDied","Data":"efb7c304c688aa9493dd13325678691a37500a7a92be47287ac52b9a39262f16"}
Oct 31 01:19:10.828008 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.827828    4553 scope.go:122] "RemoveContainer" containerID="1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"
Oct 31 01:19:10.829385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.829367    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/1.log"
Oct 31 01:19:10.829828 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.829807    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/0.log"
Oct 31 01:19:10.829924 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.829852    4553 generic.go:356] "Generic (PLEG): container finished" podID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerID="62c5c586f30875d2982ab973a09cf89d9368e9ea4b182a0aaaff39b91a86d379" exitCode=1
Oct 31 01:19:10.829924 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.829893    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerDied","Data":"62c5c586f30875d2982ab973a09cf89d9368e9ea4b182a0aaaff39b91a86d379"}
Oct 31 01:19:10.830313 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.830296    4553 scope.go:122] "RemoveContainer" containerID="38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"
Oct 31 01:19:10.837199 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.837182    4553 scope.go:122] "RemoveContainer" containerID="3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"
Oct 31 01:19:10.842350 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:10.842330    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_e2e-gw1-1c69aea164(18553d82-1ce7-45b0-a76c-c5c9755bbd76)\"" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76"
Oct 31 01:19:10.845246 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.845224    4553 scope.go:122] "RemoveContainer" containerID="2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"
Oct 31 01:19:10.852389 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.852372    4553 scope.go:122] "RemoveContainer" containerID="1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"
Oct 31 01:19:10.852644 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:10.852627    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50\": container with ID starting with 1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50 not found: ID does not exist" containerID="1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"
Oct 31 01:19:10.852721 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.852653    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50"} err="failed to get container status \"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50\": rpc error: code = NotFound desc = could not find container \"1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50\": container with ID starting with 1e0c7b672b349e98bcebc2f9923cd5e2e33a15c4054d714e3745ad779fc8ce50 not found: ID does not exist"
Oct 31 01:19:10.852721 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.852669    4553 scope.go:122] "RemoveContainer" containerID="3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"
Oct 31 01:19:10.852922 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:10.852904    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b\": container with ID starting with 3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b not found: ID does not exist" containerID="3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"
Oct 31 01:19:10.852966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.852931    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b"} err="failed to get container status \"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b\": rpc error: code = NotFound desc = could not find container \"3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b\": container with ID starting with 3f57afbf1ca00121d2ebde09de1ae1065aeb4b81adb24d701a971ffed325e53b not found: ID does not exist"
Oct 31 01:19:10.852966 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.852948    4553 scope.go:122] "RemoveContainer" containerID="2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"
Oct 31 01:19:10.853162 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:10.853145    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756\": container with ID starting with 2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756 not found: ID does not exist" containerID="2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"
Oct 31 01:19:10.853201 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.853167    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756"} err="failed to get container status \"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756\": rpc error: code = NotFound desc = could not find container \"2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756\": container with ID starting with 2e1587622fde7e7f72352ffa14e3f6d9c0ed9b0c580b02e4827d84ef71fc4756 not found: ID does not exist"
Oct 31 01:19:10.853201 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.853179    4553 scope.go:122] "RemoveContainer" containerID="38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"
Oct 31 01:19:10.853431 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:10.853414    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d\": container with ID starting with 38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d not found: ID does not exist" containerID="38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"
Oct 31 01:19:10.853483 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.853433    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d"} err="failed to get container status \"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d\": rpc error: code = NotFound desc = could not find container \"38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d\": container with ID starting with 38c2d2593ba518e784e99d20455e50595234e8f4eae6d70028067ecb9071107d not found: ID does not exist"
Oct 31 01:19:10.878630 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.878608    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:19:10.886819 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:10.886798    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-pvcfr"]
Oct 31 01:19:11.198785 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.198755    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" path="/var/lib/kubelet/pods/9813b6e9-a988-4440-baa5-30d3bbd52587/volumes"
Oct 31 01:19:11.480182 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480112    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:19:11.480406 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480395    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="storage-initializer"
Oct 31 01:19:11.480455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480408    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="storage-initializer"
Oct 31 01:19:11.480455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480434    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container"
Oct 31 01:19:11.480455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480440    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container"
Oct 31 01:19:11.480455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480448    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container"
Oct 31 01:19:11.480455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480453    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480460    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kube-rbac-proxy"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480465    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kube-rbac-proxy"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480471    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="storage-initializer"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480475    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="storage-initializer"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480482    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kube-rbac-proxy"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480487    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kube-rbac-proxy"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480528    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kserve-container"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480536    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kserve-container"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480541    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46aa7a2-9056-4c2a-8084-056f59da3083" containerName="kube-rbac-proxy"
Oct 31 01:19:11.480627 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.480547    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9813b6e9-a988-4440-baa5-30d3bbd52587" containerName="kube-rbac-proxy"
Oct 31 01:19:11.485079 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.485061    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.490575 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.490528    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:19:11.490716 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.490645    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:19:11.491565 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.491533    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:11.497586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.497564    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:19:11.497709 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.497661    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:19:11.497772 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.497726    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:19:11.497772 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.497762    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 31 01:19:11.498070 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.498050    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:19:11.498395 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.498382    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 31 01:19:11.601007 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.600976    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:19:11.604494 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.604478    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.616988 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.616964    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:19:11.626081 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.626060    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.626218 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.626086    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.626218 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.626131    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.626393 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.626263    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.626393 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.626296    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wfp\" (UniqueName: \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.643722 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.643697    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:11.698549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.698530    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 31 01:19:11.726872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.726850    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.726984 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.726879    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.726984 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.726903    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.726984 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.726969    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6d72\" (UniqueName: \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.727102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727000    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.727102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727068    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.727188 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727129    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.727228 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727209    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.727270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727255    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.727318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727282    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wfp\" (UniqueName: \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.727428 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727406    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.727577 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727555    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.727899 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.727883    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.729599 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.729579    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.742095 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.742042    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wfp\" (UniqueName: \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\") pod \"isvc-canary-stop-v2-predictor-7799b46b58-gxs56\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.803111 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.803084    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:11.828078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828046    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828197 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828117    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6d72\" (UniqueName: \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828197 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828148    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828197 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828173    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828369 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828208    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828531    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828830 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828807    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.828945 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.828861    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.831297 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.831281    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.838038 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.838020    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/1.log"
Oct 31 01:19:11.842907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.842885    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6d72\" (UniqueName: \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-sf4h7\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:11.922241 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:11.922208    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:12.082404 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.082380    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:19:12.084207 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:19:12.084184    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fb5454_9f82_4626_8ce2_ee25ef1caf38.slice/crio-dcf88e9bb945324705fb3d6da192be4dd67fa5a0f247247bd7e68fc666f82be4 WatchSource:0}: Error finding container dcf88e9bb945324705fb3d6da192be4dd67fa5a0f247247bd7e68fc666f82be4: Status 404 returned error can't find the container with id dcf88e9bb945324705fb3d6da192be4dd67fa5a0f247247bd7e68fc666f82be4
Oct 31 01:19:12.202337 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.202311    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:19:12.204375 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:19:12.204349    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe72379_d8bf_4f44_b56d_009e8928068e.slice/crio-e57258cd8794e925910ec815e7f31896004f0eb54eb83636fcd678ad14855d31 WatchSource:0}: Error finding container e57258cd8794e925910ec815e7f31896004f0eb54eb83636fcd678ad14855d31: Status 404 returned error can't find the container with id e57258cd8794e925910ec815e7f31896004f0eb54eb83636fcd678ad14855d31
Oct 31 01:19:12.843331 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.843286    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerStarted","Data":"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"}
Oct 31 01:19:12.843542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.843361    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerStarted","Data":"dcf88e9bb945324705fb3d6da192be4dd67fa5a0f247247bd7e68fc666f82be4"}
Oct 31 01:19:12.844955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.844931    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerStarted","Data":"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"}
Oct 31 01:19:12.845082 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:12.844990    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerStarted","Data":"e57258cd8794e925910ec815e7f31896004f0eb54eb83636fcd678ad14855d31"}
Oct 31 01:19:13.306084 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.306034    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:13.379202 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.379170    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:13.436345 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.436325    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/1.log"
Oct 31 01:19:13.439748 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.439730    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:13.543900 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.543864    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\" (UniqueName: \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\") pod \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") "
Oct 31 01:19:13.543900 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.543900    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\") pod \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") "
Oct 31 01:19:13.544131 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.543966    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\") pod \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") "
Oct 31 01:19:13.544131 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.543990    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\" (UniqueName: \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\") pod \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\" (UID: \"18553d82-1ce7-45b0-a76c-c5c9755bbd76\") "
Oct 31 01:19:13.544247 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.544227    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location" pod "18553d82-1ce7-45b0-a76c-c5c9755bbd76" (UID: "18553d82-1ce7-45b0-a76c-c5c9755bbd76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:13.544318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.544297    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config" pod "18553d82-1ce7-45b0-a76c-c5c9755bbd76" (UID: "18553d82-1ce7-45b0-a76c-c5c9755bbd76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:13.546240 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.546214    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls" pod "18553d82-1ce7-45b0-a76c-c5c9755bbd76" (UID: "18553d82-1ce7-45b0-a76c-c5c9755bbd76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:13.546240 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.546218    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw" pod "18553d82-1ce7-45b0-a76c-c5c9755bbd76" (UID: "18553d82-1ce7-45b0-a76c-c5c9755bbd76"). InnerVolumeSpecName "kube-api-access-pgstw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:13.644933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.644868    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18553d82-1ce7-45b0-a76c-c5c9755bbd76-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:13.644933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.644893    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:13.644933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.644904    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:13.644933 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.644913    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgstw\" (UniqueName: \"kubernetes.io/projected/18553d82-1ce7-45b0-a76c-c5c9755bbd76-kube-api-access-pgstw\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:19:13.850404 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.850376    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l_18553d82-1ce7-45b0-a76c-c5c9755bbd76/storage-initializer/1.log"
Oct 31 01:19:13.850607 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.850488    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"
Oct 31 01:19:13.850607 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.850518    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l" event={"ID":"18553d82-1ce7-45b0-a76c-c5c9755bbd76","Type":"ContainerDied","Data":"311bd5c025d3ad7c39ff50bb3c3bca8d65705601c43ec117ddb5ba03fa0b9527"}
Oct 31 01:19:13.850607 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.850584    4553 scope.go:122] "RemoveContainer" containerID="62c5c586f30875d2982ab973a09cf89d9368e9ea4b182a0aaaff39b91a86d379"
Oct 31 01:19:13.897071 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.896990    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:13.903355 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:13.903330    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-global-fail-predictor-5585b8b6f4-tjd2l"]
Oct 31 01:19:14.746741 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.746712    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747006    4553 cpu_manager.go:414] "containerMap: removing container" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747017    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747026    4553 cpu_manager.go:414] "containerMap: removing container" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747031    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747085    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.747097 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.747092    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" containerName="storage-initializer"
Oct 31 01:19:14.751508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.751489    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.757433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.757412    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 31 01:19:14.757541 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.757428    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"storage-config\""
Oct 31 01:19:14.758529 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.758513    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:19:14.758592 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.758546    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 31 01:19:14.758664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.758591    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:19:14.758664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.758613    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:19:14.758664 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.758636    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:19:14.766119 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.766099    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:19:14.772034 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.772012    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:14.851882 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.851861    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.851992 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.851891    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.851992 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.851913    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.852078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.852041    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92k6w\" (UniqueName: \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.852078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.852068    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.901580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.901554    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:14.952827 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.952801    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92k6w\" (UniqueName: \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.952909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.952830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.952909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.952853    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.952909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.952873    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.952909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.952893    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.953288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.953266    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.953530 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.953513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.953600 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.953582    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.955499 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.955478    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:14.967653 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:14.967625    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92k6w\" (UniqueName: \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:15.069274 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:15.069203    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:15.200089 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:15.200056    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18553d82-1ce7-45b0-a76c-c5c9755bbd76" path="/var/lib/kubelet/pods/18553d82-1ce7-45b0-a76c-c5c9755bbd76/volumes"
Oct 31 01:19:15.347906 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:15.347880    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:19:15.350439 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:19:15.350416    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33817769_b965_46af_ac84_6aa43aa73c26.slice/crio-ebdae3f4351ec50dcd3adc1a84a0df371099d526d6fe2e8f0d33c3acb43ced4e WatchSource:0}: Error finding container ebdae3f4351ec50dcd3adc1a84a0df371099d526d6fe2e8f0d33c3acb43ced4e: Status 404 returned error can't find the container with id ebdae3f4351ec50dcd3adc1a84a0df371099d526d6fe2e8f0d33c3acb43ced4e
Oct 31 01:19:15.861339 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:15.861300    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerStarted","Data":"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"}
Oct 31 01:19:15.861763 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:15.861363    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerStarted","Data":"ebdae3f4351ec50dcd3adc1a84a0df371099d526d6fe2e8f0d33c3acb43ced4e"}
Oct 31 01:19:16.867588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.867496    4553 generic.go:356] "Generic (PLEG): container finished" podID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerID="3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681" exitCode=0
Oct 31 01:19:16.867588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.867575    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerDied","Data":"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"}
Oct 31 01:19:16.869223 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.869200    4553 generic.go:356] "Generic (PLEG): container finished" podID="33817769-b965-46af-ac84-6aa43aa73c26" containerID="965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9" exitCode=0
Oct 31 01:19:16.869313 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.869279    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerDied","Data":"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"}
Oct 31 01:19:16.870920 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.870903    4553 generic.go:356] "Generic (PLEG): container finished" podID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerID="585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe" exitCode=0
Oct 31 01:19:16.870998 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:16.870978    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerDied","Data":"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"}
Oct 31 01:19:17.877118 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.877082    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerStarted","Data":"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"}
Oct 31 01:19:17.877621 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.877147    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerStarted","Data":"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"}
Oct 31 01:19:17.877621 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.877375    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:17.879182 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.879160    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerStarted","Data":"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"}
Oct 31 01:19:17.879292 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.879217    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerStarted","Data":"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"}
Oct 31 01:19:17.879533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.879515    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:17.879615 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.879565    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:17.881256 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.881233    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:17.881359 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.881292    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerStarted","Data":"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"}
Oct 31 01:19:17.881359 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.881328    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerStarted","Data":"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"}
Oct 31 01:19:17.881485 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.881456    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:17.903586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.903544    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podStartSLOduration=2.924420529 podStartE2EDuration="6.903532816s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.979112287s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:17.903058082 +0000 UTC m=+3020.812060736" watchObservedRunningTime="2026-10-31 01:19:17.903532816 +0000 UTC m=+3020.812535470"
Oct 31 01:19:17.929245 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.929208    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podStartSLOduration=2.889970185 podStartE2EDuration="3.929196143s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.039225958s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:17.928600225 +0000 UTC m=+3020.837602876" watchObservedRunningTime="2026-10-31 01:19:17.929196143 +0000 UTC m=+3020.838198867"
Oct 31 01:19:17.954683 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:17.954640    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podStartSLOduration=2.98953663 podStartE2EDuration="6.954625314s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965088684s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:17.953713263 +0000 UTC m=+3020.862715919" watchObservedRunningTime="2026-10-31 01:19:17.954625314 +0000 UTC m=+3020.863627969"
Oct 31 01:19:18.885976 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:18.885938    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:18.886450 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:18.886214    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:18.886450 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:18.886266    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:18.886917 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:18.886895    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:18.886998 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:18.886980    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:19.889396 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:19.889350    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:19.889887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:19.889454    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:23.890766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:23.890736    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:19:23.891327 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:23.891248    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:24.894023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:24.893994    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:19:24.894431 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:24.894075    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:19:24.894535 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:24.894506    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:24.894656 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:24.894539    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:27.130138 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:27.130106    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:19:33.891797 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:33.891747    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:34.895454 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:34.895417    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:34.895898 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:34.895422    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:43.891287 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:43.891250    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:44.894468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:44.894405    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:44.894925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:44.894422    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:47.837012 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:47.836982    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:53.892122 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:53.892086    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:19:54.894613 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:54.894535    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:19:54.895169 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:19:54.894547    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:19:57.131029 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:19:57.130993    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:20:03.895638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:03.895593    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:20:04.894416 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:04.894377    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:20:04.894623 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:04.894466    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:20:13.892009 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:13.891965    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:20:14.895152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:14.895108    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:20:14.895152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:14.895126    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:20:23.892031 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:23.891992    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:20:24.895111 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:24.895073    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:20:24.895558 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:24.895073    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:20:27.129484 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:27.129447    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:20:28.195173 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:28.195130    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 31 01:20:29.200328 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:29.200290    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:20:34.895341 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:34.895304    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 31 01:20:37.200137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:37.200104    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:20:38.195848 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:38.195815    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:20:39.201404 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:39.201373    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:20:41.645247 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.645211    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:20:41.645644 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.645468    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container" containerID="cri-o://87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a" gracePeriod=30
Oct 31 01:20:41.645644 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.645587    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kube-rbac-proxy" containerID="cri-o://1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77" gracePeriod=30
Oct 31 01:20:41.708086 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.708051    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:20:41.708338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.708312    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kube-rbac-proxy" containerID="cri-o://07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214" gracePeriod=30
Oct 31 01:20:41.708428 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.708298    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container" containerID="cri-o://c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93" gracePeriod=30
Oct 31 01:20:41.714160 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.714138    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:41.862938 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:41.862898    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:42.219626 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:42.219593    4553 generic.go:356] "Generic (PLEG): container finished" podID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerID="1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77" exitCode=0
Oct 31 01:20:42.219855 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:42.219670    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerDied","Data":"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"}
Oct 31 01:20:42.221778 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:42.221753    4553 generic.go:356] "Generic (PLEG): container finished" podID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerID="07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214" exitCode=0
Oct 31 01:20:42.221878 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:42.221817    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerDied","Data":"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"}
Oct 31 01:20:43.886400 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:43.886355    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Oct 31 01:20:44.890468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:44.890424    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Oct 31 01:20:44.928697 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:44.928653    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:20:44.928949 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:44.928922    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" containerID="cri-o://2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3" gracePeriod=30
Oct 31 01:20:44.929037 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:44.928947    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kube-rbac-proxy" containerID="cri-o://0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2" gracePeriod=30
Oct 31 01:20:44.994653 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:44.994618    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:45.235540 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:45.235509    4553 generic.go:356] "Generic (PLEG): container finished" podID="33817769-b965-46af-ac84-6aa43aa73c26" containerID="0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2" exitCode=0
Oct 31 01:20:45.235734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:45.235581    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerDied","Data":"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"}
Oct 31 01:20:45.738795 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:45.738762    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:46.183703 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:46.183652    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:46.384297 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:46.384256    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:47.088216 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.088191    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:20:47.153421 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.153396    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:20:47.204391 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204367    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\" (UniqueName: \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\") pod \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") "
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204415    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\") pod \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") "
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204445    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\") pod \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") "
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204475    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\") pod \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") "
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204493    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\" (UniqueName: \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\") pod \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\" (UID: \"90fb5454-9f82-4626-8ce2-ee25ef1caf38\") "
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204809    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert" pod "90fb5454-9f82-4626-8ce2-ee25ef1caf38" (UID: "90fb5454-9f82-4626-8ce2-ee25ef1caf38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:47.204835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204823    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location" pod "90fb5454-9f82-4626-8ce2-ee25ef1caf38" (UID: "90fb5454-9f82-4626-8ce2-ee25ef1caf38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:47.205061 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.204828    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config" pod "90fb5454-9f82-4626-8ce2-ee25ef1caf38" (UID: "90fb5454-9f82-4626-8ce2-ee25ef1caf38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:47.206602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.206584    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls" pod "90fb5454-9f82-4626-8ce2-ee25ef1caf38" (UID: "90fb5454-9f82-4626-8ce2-ee25ef1caf38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:47.206725 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.206705    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp" pod "90fb5454-9f82-4626-8ce2-ee25ef1caf38" (UID: "90fb5454-9f82-4626-8ce2-ee25ef1caf38"). InnerVolumeSpecName "kube-api-access-28wfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:47.245024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.245000    4553 generic.go:356] "Generic (PLEG): container finished" podID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerID="87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a" exitCode=0
Oct 31 01:20:47.245137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.245057    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"
Oct 31 01:20:47.245137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.245075    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerDied","Data":"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"}
Oct 31 01:20:47.245137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.245129    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56" event={"ID":"90fb5454-9f82-4626-8ce2-ee25ef1caf38","Type":"ContainerDied","Data":"dcf88e9bb945324705fb3d6da192be4dd67fa5a0f247247bd7e68fc666f82be4"}
Oct 31 01:20:47.245247 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.245156    4553 scope.go:122] "RemoveContainer" containerID="1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"
Oct 31 01:20:47.247274 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.247248    4553 generic.go:356] "Generic (PLEG): container finished" podID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerID="c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93" exitCode=0
Oct 31 01:20:47.247399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.247309    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"
Oct 31 01:20:47.247399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.247316    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerDied","Data":"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"}
Oct 31 01:20:47.247399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.247361    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7" event={"ID":"2fe72379-d8bf-4f44-b56d-009e8928068e","Type":"ContainerDied","Data":"e57258cd8794e925910ec815e7f31896004f0eb54eb83636fcd678ad14855d31"}
Oct 31 01:20:47.254449 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.254275    4553 scope.go:122] "RemoveContainer" containerID="87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"
Oct 31 01:20:47.262140 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.262120    4553 scope.go:122] "RemoveContainer" containerID="3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"
Oct 31 01:20:47.269236 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.269217    4553 scope.go:122] "RemoveContainer" containerID="1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"
Oct 31 01:20:47.269483 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.269467    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77\": container with ID starting with 1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77 not found: ID does not exist" containerID="1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"
Oct 31 01:20:47.269537 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.269491    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77"} err="failed to get container status \"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77\": rpc error: code = NotFound desc = could not find container \"1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77\": container with ID starting with 1393d8240083b517a2a627cb9ec375ceaf4e48ecad1dc51933a9604df8f67b77 not found: ID does not exist"
Oct 31 01:20:47.269537 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.269505    4553 scope.go:122] "RemoveContainer" containerID="87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"
Oct 31 01:20:47.269759 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.269739    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a\": container with ID starting with 87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a not found: ID does not exist" containerID="87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"
Oct 31 01:20:47.269826 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.269761    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a"} err="failed to get container status \"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a\": rpc error: code = NotFound desc = could not find container \"87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a\": container with ID starting with 87597131346a2ce3d00b83597e69ea34e51bd40741620b69748b36e208e3813a not found: ID does not exist"
Oct 31 01:20:47.269826 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.269774    4553 scope.go:122] "RemoveContainer" containerID="3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"
Oct 31 01:20:47.270004 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.269990    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681\": container with ID starting with 3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681 not found: ID does not exist" containerID="3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"
Oct 31 01:20:47.270042 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.270007    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681"} err="failed to get container status \"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681\": rpc error: code = NotFound desc = could not find container \"3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681\": container with ID starting with 3c4bceae386ac0064ecfefc97703ad82eeb5fa99b20e4613453cc2bcd1af2681 not found: ID does not exist"
Oct 31 01:20:47.270042 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.270019    4553 scope.go:122] "RemoveContainer" containerID="07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"
Oct 31 01:20:47.274544 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.274523    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:20:47.277238 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.277222    4553 scope.go:122] "RemoveContainer" containerID="c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"
Oct 31 01:20:47.280645 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.280623    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-v2-predictor-7799b46b58-gxs56"]
Oct 31 01:20:47.284399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.284383    4553 scope.go:122] "RemoveContainer" containerID="585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"
Oct 31 01:20:47.291245 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.291229    4553 scope.go:122] "RemoveContainer" containerID="07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"
Oct 31 01:20:47.291514 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.291494    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214\": container with ID starting with 07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214 not found: ID does not exist" containerID="07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"
Oct 31 01:20:47.291564 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.291522    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214"} err="failed to get container status \"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214\": rpc error: code = NotFound desc = could not find container \"07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214\": container with ID starting with 07bc6b2d4b6edfedf6223f92b2fc00501dc9347370c35cd65e968ffa776a9214 not found: ID does not exist"
Oct 31 01:20:47.291564 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.291539    4553 scope.go:122] "RemoveContainer" containerID="c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"
Oct 31 01:20:47.291881 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.291863    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93\": container with ID starting with c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93 not found: ID does not exist" containerID="c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"
Oct 31 01:20:47.291955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.291889    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93"} err="failed to get container status \"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93\": rpc error: code = NotFound desc = could not find container \"c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93\": container with ID starting with c8d44666376f55da98d37418675c1c082dcc5ea47609b31f5a86bb32139f0e93 not found: ID does not exist"
Oct 31 01:20:47.291955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.291910    4553 scope.go:122] "RemoveContainer" containerID="585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"
Oct 31 01:20:47.292162 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:47.292147    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe\": container with ID starting with 585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe not found: ID does not exist" containerID="585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"
Oct 31 01:20:47.292204 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.292169    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe"} err="failed to get container status \"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe\": rpc error: code = NotFound desc = could not find container \"585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe\": container with ID starting with 585e2a5446aaa72d566604716fd2019de2442fa68922343eb8b33ec1c6e9bffe not found: ID does not exist"
Oct 31 01:20:47.305437 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305413    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\" (UniqueName: \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\") pod \"2fe72379-d8bf-4f44-b56d-009e8928068e\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") "
Oct 31 01:20:47.305517 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305493    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\") pod \"2fe72379-d8bf-4f44-b56d-009e8928068e\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") "
Oct 31 01:20:47.305586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305559    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\") pod \"2fe72379-d8bf-4f44-b56d-009e8928068e\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") "
Oct 31 01:20:47.305642 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305594    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\") pod \"2fe72379-d8bf-4f44-b56d-009e8928068e\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") "
Oct 31 01:20:47.305642 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305626    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\") pod \"2fe72379-d8bf-4f44-b56d-009e8928068e\" (UID: \"2fe72379-d8bf-4f44-b56d-009e8928068e\") "
Oct 31 01:20:47.305842 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305818    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location" pod "2fe72379-d8bf-4f44-b56d-009e8928068e" (UID: "2fe72379-d8bf-4f44-b56d-009e8928068e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:47.305907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305886    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.305970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305905    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.305970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305938    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.305970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305939    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert" pod "2fe72379-d8bf-4f44-b56d-009e8928068e" (UID: "2fe72379-d8bf-4f44-b56d-009e8928068e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:47.305970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305945    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config" pod "2fe72379-d8bf-4f44-b56d-009e8928068e" (UID: "2fe72379-d8bf-4f44-b56d-009e8928068e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:47.305970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305955    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wfp\" (UniqueName: \"kubernetes.io/projected/90fb5454-9f82-4626-8ce2-ee25ef1caf38-kube-api-access-28wfp\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.306222 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.305984    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe72379-d8bf-4f44-b56d-009e8928068e-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.306222 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.306002    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90fb5454-9f82-4626-8ce2-ee25ef1caf38-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.307559 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.307543    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72" pod "2fe72379-d8bf-4f44-b56d-009e8928068e" (UID: "2fe72379-d8bf-4f44-b56d-009e8928068e"). InnerVolumeSpecName "kube-api-access-f6d72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:47.307809 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.307794    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls" pod "2fe72379-d8bf-4f44-b56d-009e8928068e" (UID: "2fe72379-d8bf-4f44-b56d-009e8928068e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:47.407034 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.406997    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.407034 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.407031    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.407230 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.407045    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fe72379-d8bf-4f44-b56d-009e8928068e-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.407230 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.407056    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6d72\" (UniqueName: \"kubernetes.io/projected/2fe72379-d8bf-4f44-b56d-009e8928068e-kube-api-access-f6d72\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:47.577733 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.573552    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:20:47.581638 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.581599    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-sf4h7"]
Oct 31 01:20:47.883241 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:47.883207    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:48.287029 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.286990    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287416    4553 cpu_manager.go:414] "containerMap: removing container" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container"
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287463    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container"
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287478    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container"
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287485    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container"
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287494    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kube-rbac-proxy"
Oct 31 01:20:48.287507 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287500    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kube-rbac-proxy"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287515    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="storage-initializer"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287522    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="storage-initializer"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287552    4553 cpu_manager.go:414] "containerMap: removing container" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kube-rbac-proxy"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287559    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kube-rbac-proxy"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287569    4553 cpu_manager.go:414] "containerMap: removing container" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="storage-initializer"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287577    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="storage-initializer"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287668    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kube-rbac-proxy"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287702    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" containerName="kserve-container"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287710    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kserve-container"
Oct 31 01:20:48.287887 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.287717    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" containerName="kube-rbac-proxy"
Oct 31 01:20:48.292042 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.292013    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:48.292588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.292574    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.299476 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.299456    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 31 01:20:48.302060 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.302037    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:20:48.303045 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.303029    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:20:48.303321 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.303303    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 31 01:20:48.304468 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.304407    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:20:48.305954 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.305935    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:20:48.399936 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.399899    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:20:48.400108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.399988    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:20:48.416185 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.416159    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf948\" (UniqueName: \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.416299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.416199    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.416299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.416223    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.416384 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.416323    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.416384 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.416345    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.432013 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.431982    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:48.517647 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.517613    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf948\" (UniqueName: \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.517834 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.517661    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.517903 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.517850    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.517947 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.517924    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.517999 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.517950    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.518281 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.518259    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.518417 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.518395    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.518462 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.518417    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.520453 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.520436    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.532319 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.532295    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf948\" (UniqueName: \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\") pod \"isvc-lightgbm-predictor-85d97887df-kgb7v\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.609777 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.609700    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:20:48.887277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:48.887252    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:20:48.889727 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:20:48.889701    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a194fe_cef4_4530_af2e_aaba29300035.slice/crio-bf7e1adce2a99d9f24e500f41f930aa58e51825492d96520884e030e088acb64 WatchSource:0}: Error finding container bf7e1adce2a99d9f24e500f41f930aa58e51825492d96520884e030e088acb64: Status 404 returned error can't find the container with id bf7e1adce2a99d9f24e500f41f930aa58e51825492d96520884e030e088acb64
Oct 31 01:20:49.199952 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.199915    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe72379-d8bf-4f44-b56d-009e8928068e" path="/var/lib/kubelet/pods/2fe72379-d8bf-4f44-b56d-009e8928068e/volumes"
Oct 31 01:20:49.200652 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.200632    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fb5454-9f82-4626-8ce2-ee25ef1caf38" path="/var/lib/kubelet/pods/90fb5454-9f82-4626-8ce2-ee25ef1caf38/volumes"
Oct 31 01:20:49.200749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.200723    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Oct 31 01:20:49.259590 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.259553    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerStarted","Data":"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"}
Oct 31 01:20:49.259780 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.259609    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerStarted","Data":"bf7e1adce2a99d9f24e500f41f930aa58e51825492d96520884e030e088acb64"}
Oct 31 01:20:49.890591 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:49.890551    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Oct 31 01:20:50.074773 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.074750    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:20:50.129592 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.129511    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\" (UniqueName: \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\") pod \"33817769-b965-46af-ac84-6aa43aa73c26\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") "
Oct 31 01:20:50.129592 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.129553    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\") pod \"33817769-b965-46af-ac84-6aa43aa73c26\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") "
Oct 31 01:20:50.129854 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.129593    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\" (UniqueName: \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\") pod \"33817769-b965-46af-ac84-6aa43aa73c26\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") "
Oct 31 01:20:50.129854 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.129655    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\") pod \"33817769-b965-46af-ac84-6aa43aa73c26\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") "
Oct 31 01:20:50.129854 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.129738    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\") pod \"33817769-b965-46af-ac84-6aa43aa73c26\" (UID: \"33817769-b965-46af-ac84-6aa43aa73c26\") "
Oct 31 01:20:50.130052 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.130024    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location" pod "33817769-b965-46af-ac84-6aa43aa73c26" (UID: "33817769-b965-46af-ac84-6aa43aa73c26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:50.130052 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.130032    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config" pod "33817769-b965-46af-ac84-6aa43aa73c26" (UID: "33817769-b965-46af-ac84-6aa43aa73c26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:50.130223 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.130125    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert" pod "33817769-b965-46af-ac84-6aa43aa73c26" (UID: "33817769-b965-46af-ac84-6aa43aa73c26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:50.131975 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.131954    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w" pod "33817769-b965-46af-ac84-6aa43aa73c26" (UID: "33817769-b965-46af-ac84-6aa43aa73c26"). InnerVolumeSpecName "kube-api-access-92k6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:50.132101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.132081    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls" pod "33817769-b965-46af-ac84-6aa43aa73c26" (UID: "33817769-b965-46af-ac84-6aa43aa73c26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:50.230958 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.230919    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92k6w\" (UniqueName: \"kubernetes.io/projected/33817769-b965-46af-ac84-6aa43aa73c26-kube-api-access-92k6w\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:50.230958 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.230951    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:50.230958 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.230962    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33817769-b965-46af-ac84-6aa43aa73c26-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:50.231171 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.230972    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33817769-b965-46af-ac84-6aa43aa73c26-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:50.231171 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.230982    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33817769-b965-46af-ac84-6aa43aa73c26-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:50.265842 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.265804    4553 generic.go:356] "Generic (PLEG): container finished" podID="33817769-b965-46af-ac84-6aa43aa73c26" containerID="2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3" exitCode=0
Oct 31 01:20:50.265986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.265859    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"
Oct 31 01:20:50.265986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.265893    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerDied","Data":"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"}
Oct 31 01:20:50.265986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.265953    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6" event={"ID":"33817769-b965-46af-ac84-6aa43aa73c26","Type":"ContainerDied","Data":"ebdae3f4351ec50dcd3adc1a84a0df371099d526d6fe2e8f0d33c3acb43ced4e"}
Oct 31 01:20:50.265986 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.265980    4553 scope.go:122] "RemoveContainer" containerID="0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"
Oct 31 01:20:50.274859 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.274841    4553 scope.go:122] "RemoveContainer" containerID="2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"
Oct 31 01:20:50.282456 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.282438    4553 scope.go:122] "RemoveContainer" containerID="965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"
Oct 31 01:20:50.289615 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.289597    4553 scope.go:122] "RemoveContainer" containerID="0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"
Oct 31 01:20:50.289976 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:50.289948    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2\": container with ID starting with 0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2 not found: ID does not exist" containerID="0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"
Oct 31 01:20:50.290074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.289983    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2"} err="failed to get container status \"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2\": rpc error: code = NotFound desc = could not find container \"0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2\": container with ID starting with 0c407aca71fa052025200e4db862f98e8ec6906deee9485f66eabaefb7dbf8b2 not found: ID does not exist"
Oct 31 01:20:50.290074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.290001    4553 scope.go:122] "RemoveContainer" containerID="2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"
Oct 31 01:20:50.290405 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:50.290288    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3\": container with ID starting with 2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3 not found: ID does not exist" containerID="2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"
Oct 31 01:20:50.290405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.290316    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3"} err="failed to get container status \"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3\": rpc error: code = NotFound desc = could not find container \"2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3\": container with ID starting with 2b183db8538f27c90a0ffa2424ccaf9a5e571998c47289360ec8b092848d47f3 not found: ID does not exist"
Oct 31 01:20:50.290405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.290334    4553 scope.go:122] "RemoveContainer" containerID="965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"
Oct 31 01:20:50.290627 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:50.290603    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9\": container with ID starting with 965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9 not found: ID does not exist" containerID="965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"
Oct 31 01:20:50.290702 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.290635    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9"} err="failed to get container status \"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9\": rpc error: code = NotFound desc = could not find container \"965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9\": container with ID starting with 965fffa1a7f6e9c2e588d61f58cbea13fa73596539ad77f990b0d85ef68552f9 not found: ID does not exist"
Oct 31 01:20:50.291908 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.291890    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:20:50.300144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:50.300121    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-pass-predictor-c769bc97d-wf9b6"]
Oct 31 01:20:51.199404 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:51.199368    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33817769-b965-46af-ac84-6aa43aa73c26" path="/var/lib/kubelet/pods/33817769-b965-46af-ac84-6aa43aa73c26/volumes"
Oct 31 01:20:53.280264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:53.280231    4553 generic.go:356] "Generic (PLEG): container finished" podID="77a194fe-cef4-4530-af2e-aaba29300035" containerID="f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453" exitCode=0
Oct 31 01:20:53.280700 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:53.280275    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerDied","Data":"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"}
Oct 31 01:20:53.281482 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:53.281463    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:20:56.263459 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:56.263423    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:57.133133 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:20:57.133056    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:20:57.574310 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.573869    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:20:57.574310 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574289    4553 cpu_manager.go:414] "containerMap: removing container" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="storage-initializer"
Oct 31 01:20:57.574310 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574303    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="storage-initializer"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574325    4553 cpu_manager.go:414] "containerMap: removing container" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kube-rbac-proxy"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574330    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kube-rbac-proxy"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574341    4553 cpu_manager.go:414] "containerMap: removing container" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574346    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574392    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kube-rbac-proxy"
Oct 31 01:20:57.574807 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.574401    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="33817769-b965-46af-ac84-6aa43aa73c26" containerName="kserve-container"
Oct 31 01:20:57.591401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.591334    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:20:57.591634 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.591551    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.597584 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.597555    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:57.600309 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.600289    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"storage-config\""
Oct 31 01:20:57.601320 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601071    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 31 01:20:57.601320 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601158    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:20:57.601320 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601286    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:20:57.601455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601363    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:20:57.601455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601403    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 31 01:20:57.601455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.601434    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:20:57.690078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.690035    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.690244 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.690089    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.690244 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.690196    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrqr\" (UniqueName: \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.690395 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.690239    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.690395 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.690354    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.740797 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.740760    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:57.791267 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791229    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrqr\" (UniqueName: \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.791480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791286    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.791480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791328    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.791480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791408    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.791480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791437    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.792019 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.791991    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.792156 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.792134    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.792225 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.792203    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.794243 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.794218    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.806566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.806535    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrqr\" (UniqueName: \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:57.917381 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:57.917345    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:20:58.210078 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:58.210042    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:20:58.212019 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:20:58.211982    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b90f612_d2ac_457a_b7a5_040df3902b50.slice/crio-35defb984d3f866f3083e550a54a01285f72b23ffa628cc366651a25d641d54f WatchSource:0}: Error finding container 35defb984d3f866f3083e550a54a01285f72b23ffa628cc366651a25d641d54f: Status 404 returned error can't find the container with id 35defb984d3f866f3083e550a54a01285f72b23ffa628cc366651a25d641d54f
Oct 31 01:20:58.309043 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:58.308929    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerStarted","Data":"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"}
Oct 31 01:20:58.309043 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:20:58.309003    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerStarted","Data":"35defb984d3f866f3083e550a54a01285f72b23ffa628cc366651a25d641d54f"}
Oct 31 01:21:00.319183 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.319091    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerStarted","Data":"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"}
Oct 31 01:21:00.319183 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.319158    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerStarted","Data":"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"}
Oct 31 01:21:00.319633 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.319358    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:21:00.320720 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.320696    4553 generic.go:356] "Generic (PLEG): container finished" podID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerID="9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f" exitCode=0
Oct 31 01:21:00.320818 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.320700    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerDied","Data":"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"}
Oct 31 01:21:00.348354 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:00.348308    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podStartSLOduration=1.866713056 podStartE2EDuration="12.348295518s" totalImagesPullingTime="6.712085702s" totalInitContainerRuntime="3.76949676s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:00.346401598 +0000 UTC m=+3123.255404253" watchObservedRunningTime="2026-10-31 01:21:00.348295518 +0000 UTC m=+3123.257298172"
Oct 31 01:21:01.327205 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.327169    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerStarted","Data":"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"}
Oct 31 01:21:01.327601 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.327231    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerStarted","Data":"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"}
Oct 31 01:21:01.327601 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.327434    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:21:01.327721 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.327637    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:21:01.327721 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.327712    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:21:01.328970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.328942    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:01.328970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.328958    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:01.357416 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:01.357366    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podStartSLOduration=3.083570246 podStartE2EDuration="4.357353535s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.273783289s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:01.356796274 +0000 UTC m=+3124.265798929" watchObservedRunningTime="2026-10-31 01:21:01.357353535 +0000 UTC m=+3124.266356188"
Oct 31 01:21:02.331345 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:02.331297    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:02.331758 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:02.331430    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:07.336484 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:07.336452    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:21:07.337005 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:07.336910    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:21:07.337066 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:07.336994    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:07.337407 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:07.337387    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:17.337510 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:17.337472    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:17.338044 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:17.337473    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:27.129910 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:21:27.129866    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:21:27.337580 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:27.337539    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:27.337818 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:27.337539    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:37.336884 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:37.336846    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:37.337321 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:37.337286    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:47.337503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:47.337457    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:47.337503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:47.337470    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:21:47.960861 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:47.960824    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:57.129863 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:21:57.129829    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:21:57.337643 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:57.337603    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:21:57.337870 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:21:57.337603    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:22:07.337532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:07.337483    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 31 01:22:07.337963 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:07.337528    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:22:17.337856 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.337772    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:22:17.338389 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.338371    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:22:17.717774 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.717745    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:22:17.717980 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.717961    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container" containerID="cri-o://5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61" gracePeriod=30
Oct 31 01:22:17.718076 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.718049    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kube-rbac-proxy" containerID="cri-o://aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841" gracePeriod=30
Oct 31 01:22:17.781661 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:17.781628    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:18.632828 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.632790    4553 generic.go:356] "Generic (PLEG): container finished" podID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerID="aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841" exitCode=0
Oct 31 01:22:18.633206 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.632851    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerDied","Data":"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"}
Oct 31 01:22:18.874855 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.874819    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:18.878317 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.878300    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:18.884322 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.884252    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Oct 31 01:22:18.884448 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.884388    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Oct 31 01:22:18.891394 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.891371    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:18.941916 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.941880    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:18.941916 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.941926    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8pd\" (UniqueName: \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:18.942170 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.941951    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:18.942170 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:18.942092    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.042760 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.042725    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.042760 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.042762    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8pd\" (UniqueName: \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.042978 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.042785    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.042978 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.042818    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.043249 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.043226    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.043502 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.043484    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.045426 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.045406    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.058098 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.058066    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8pd\" (UniqueName: \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.195457 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.195422    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:19.364300 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.364268    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:19.475740 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.475635    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:19.481645 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:22:19.481618    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b66f70e_6199_4322_bab0_74e46a86b1d9.slice/crio-4354c1be01aaee5825c31e601c9d1ce3c7afae30907a7d8d9ad9780dd66d2156 WatchSource:0}: Error finding container 4354c1be01aaee5825c31e601c9d1ce3c7afae30907a7d8d9ad9780dd66d2156: Status 404 returned error can't find the container with id 4354c1be01aaee5825c31e601c9d1ce3c7afae30907a7d8d9ad9780dd66d2156
Oct 31 01:22:19.638204 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.638162    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerStarted","Data":"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"}
Oct 31 01:22:19.638629 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:19.638248    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerStarted","Data":"4354c1be01aaee5825c31e601c9d1ce3c7afae30907a7d8d9ad9780dd66d2156"}
Oct 31 01:22:22.331605 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:22.331557    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Oct 31 01:22:22.951703 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:22.951659    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:22:23.076760 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.076642    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\" (UniqueName: \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\") pod \"6b90f612-d2ac-457a-b7a5-040df3902b50\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") "
Oct 31 01:22:23.076946 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.076755    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\") pod \"6b90f612-d2ac-457a-b7a5-040df3902b50\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") "
Oct 31 01:22:23.076946 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.076800    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\") pod \"6b90f612-d2ac-457a-b7a5-040df3902b50\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") "
Oct 31 01:22:23.076946 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.076920    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\") pod \"6b90f612-d2ac-457a-b7a5-040df3902b50\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") "
Oct 31 01:22:23.077131 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.076967    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\") pod \"6b90f612-d2ac-457a-b7a5-040df3902b50\" (UID: \"6b90f612-d2ac-457a-b7a5-040df3902b50\") "
Oct 31 01:22:23.077131 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.077112    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert" pod "6b90f612-d2ac-457a-b7a5-040df3902b50" (UID: "6b90f612-d2ac-457a-b7a5-040df3902b50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:23.077251 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.077233    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:23.077308 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.077282    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location" pod "6b90f612-d2ac-457a-b7a5-040df3902b50" (UID: "6b90f612-d2ac-457a-b7a5-040df3902b50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:23.077461 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.077374    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config" pod "6b90f612-d2ac-457a-b7a5-040df3902b50" (UID: "6b90f612-d2ac-457a-b7a5-040df3902b50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:23.079048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.079022    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls" pod "6b90f612-d2ac-457a-b7a5-040df3902b50" (UID: "6b90f612-d2ac-457a-b7a5-040df3902b50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:23.079048 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.079038    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr" pod "6b90f612-d2ac-457a-b7a5-040df3902b50" (UID: "6b90f612-d2ac-457a-b7a5-040df3902b50"). InnerVolumeSpecName "kube-api-access-gjrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:23.177602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.177565    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b90f612-d2ac-457a-b7a5-040df3902b50-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:23.177602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.177595    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:23.177602 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.177607    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrqr\" (UniqueName: \"kubernetes.io/projected/6b90f612-d2ac-457a-b7a5-040df3902b50-kube-api-access-gjrqr\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:23.177850 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.177620    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b90f612-d2ac-457a-b7a5-040df3902b50-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:23.657372 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.657341    4553 generic.go:356] "Generic (PLEG): container finished" podID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerID="5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61" exitCode=0
Oct 31 01:22:23.657886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.657377    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerDied","Data":"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"}
Oct 31 01:22:23.657886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.657414    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"
Oct 31 01:22:23.657886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.657426    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz" event={"ID":"6b90f612-d2ac-457a-b7a5-040df3902b50","Type":"ContainerDied","Data":"35defb984d3f866f3083e550a54a01285f72b23ffa628cc366651a25d641d54f"}
Oct 31 01:22:23.657886 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.657452    4553 scope.go:122] "RemoveContainer" containerID="aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"
Oct 31 01:22:23.665549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.665531    4553 scope.go:122] "RemoveContainer" containerID="5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"
Oct 31 01:22:23.672963 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.672944    4553 scope.go:122] "RemoveContainer" containerID="9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"
Oct 31 01:22:23.680060 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680043    4553 scope.go:122] "RemoveContainer" containerID="aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"
Oct 31 01:22:23.680302 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:23.680284    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841\": container with ID starting with aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841 not found: ID does not exist" containerID="aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"
Oct 31 01:22:23.680357 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680313    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841"} err="failed to get container status \"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841\": rpc error: code = NotFound desc = could not find container \"aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841\": container with ID starting with aa9107aa14d6ac8f5d3436f328e471471bb34cfeb2854d0d329f879d038de841 not found: ID does not exist"
Oct 31 01:22:23.680357 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680329    4553 scope.go:122] "RemoveContainer" containerID="5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"
Oct 31 01:22:23.680591 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:23.680572    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61\": container with ID starting with 5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61 not found: ID does not exist" containerID="5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"
Oct 31 01:22:23.680636 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680600    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61"} err="failed to get container status \"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61\": rpc error: code = NotFound desc = could not find container \"5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61\": container with ID starting with 5e6bded5243d6bcea5b9285525e13b341492351d0d4958f7cd6cd56ad7324e61 not found: ID does not exist"
Oct 31 01:22:23.680636 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680615    4553 scope.go:122] "RemoveContainer" containerID="9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"
Oct 31 01:22:23.680894 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:23.680856    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f\": container with ID starting with 9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f not found: ID does not exist" containerID="9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"
Oct 31 01:22:23.680894 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.680887    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f"} err="failed to get container status \"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f\": rpc error: code = NotFound desc = could not find container \"9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f\": container with ID starting with 9cadd07b36ea9748254e92714e09eb4b36cc9796f20f6c1195f94bedd161847f not found: ID does not exist"
Oct 31 01:22:23.682672 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.682652    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:22:23.691130 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:23.691110    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-pass-predictor-54f9f444dc-xsvbz"]
Oct 31 01:22:24.662968 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:24.662943    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/0.log"
Oct 31 01:22:24.663405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:24.662979    4553 generic.go:356] "Generic (PLEG): container finished" podID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerID="237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d" exitCode=1
Oct 31 01:22:24.663405 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:24.663052    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerDied","Data":"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"}
Oct 31 01:22:25.195384 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:25.195343    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:22:25.205642 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:25.205604    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" path="/var/lib/kubelet/pods/6b90f612-d2ac-457a-b7a5-040df3902b50/volumes"
Oct 31 01:22:25.669722 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:25.669672    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/0.log"
Oct 31 01:22:25.670126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:25.669809    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerStarted","Data":"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"}
Oct 31 01:22:27.129574 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:27.129536    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:22:28.883663 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:28.883633    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:28.884231 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:28.883863    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer" containerID="cri-o://d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c" gracePeriod=30
Oct 31 01:22:28.949006 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:28.948971    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:30.797338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:30.797304    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:31.627260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.627237    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/1.log"
Oct 31 01:22:31.627669 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.627653    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/0.log"
Oct 31 01:22:31.629982 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.629966    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:31.696482 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.696459    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/1.log"
Oct 31 01:22:31.696827 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.696812    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m_0b66f70e-6199-4322-bab0-74e46a86b1d9/storage-initializer/0.log"
Oct 31 01:22:31.696879 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.696848    4553 generic.go:356] "Generic (PLEG): container finished" podID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerID="d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c" exitCode=1
Oct 31 01:22:31.696930 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.696923    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"
Oct 31 01:22:31.696972 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.696928    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerDied","Data":"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"}
Oct 31 01:22:31.697030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.697015    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m" event={"ID":"0b66f70e-6199-4322-bab0-74e46a86b1d9","Type":"ContainerDied","Data":"4354c1be01aaee5825c31e601c9d1ce3c7afae30907a7d8d9ad9780dd66d2156"}
Oct 31 01:22:31.697067 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.697060    4553 scope.go:122] "RemoveContainer" containerID="d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"
Oct 31 01:22:31.705219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.705201    4553 scope.go:122] "RemoveContainer" containerID="237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"
Oct 31 01:22:31.712326 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.712304    4553 scope.go:122] "RemoveContainer" containerID="d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"
Oct 31 01:22:31.712602 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:31.712571    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c\": container with ID starting with d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c not found: ID does not exist" containerID="d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"
Oct 31 01:22:31.712651 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.712611    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c"} err="failed to get container status \"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c\": rpc error: code = NotFound desc = could not find container \"d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c\": container with ID starting with d664a3ff208a9f2eafe9c49ed6dd20f7c5a1359b92e13d75a9c3398509b9927c not found: ID does not exist"
Oct 31 01:22:31.712651 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.712626    4553 scope.go:122] "RemoveContainer" containerID="237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"
Oct 31 01:22:31.713121 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:31.712865    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d\": container with ID starting with 237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d not found: ID does not exist" containerID="237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"
Oct 31 01:22:31.713121 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.712905    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d"} err="failed to get container status \"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d\": rpc error: code = NotFound desc = could not find container \"237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d\": container with ID starting with 237b30123c224f730caf0b148e9e04ba2310b7e5d88e2b8c19cfedd86577cb9d not found: ID does not exist"
Oct 31 01:22:31.750682 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.750657    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\" (UniqueName: \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\") pod \"0b66f70e-6199-4322-bab0-74e46a86b1d9\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") "
Oct 31 01:22:31.750802 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.750748    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\") pod \"0b66f70e-6199-4322-bab0-74e46a86b1d9\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") "
Oct 31 01:22:31.750871 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.750800    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\") pod \"0b66f70e-6199-4322-bab0-74e46a86b1d9\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") "
Oct 31 01:22:31.750928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.750894    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\") pod \"0b66f70e-6199-4322-bab0-74e46a86b1d9\" (UID: \"0b66f70e-6199-4322-bab0-74e46a86b1d9\") "
Oct 31 01:22:31.751049 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.751027    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location" pod "0b66f70e-6199-4322-bab0-74e46a86b1d9" (UID: "0b66f70e-6199-4322-bab0-74e46a86b1d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:31.751133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.751113    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b66f70e-6199-4322-bab0-74e46a86b1d9-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:31.751235 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.751214    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config" pod "0b66f70e-6199-4322-bab0-74e46a86b1d9" (UID: "0b66f70e-6199-4322-bab0-74e46a86b1d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:31.752921 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.752899    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls" pod "0b66f70e-6199-4322-bab0-74e46a86b1d9" (UID: "0b66f70e-6199-4322-bab0-74e46a86b1d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:31.752998 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.752938    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd" pod "0b66f70e-6199-4322-bab0-74e46a86b1d9" (UID: "0b66f70e-6199-4322-bab0-74e46a86b1d9"). InnerVolumeSpecName "kube-api-access-tj8pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:31.852470 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.852423    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b66f70e-6199-4322-bab0-74e46a86b1d9-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:31.852470 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.852467    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:31.852470 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:31.852482    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8pd\" (UniqueName: \"kubernetes.io/projected/0b66f70e-6199-4322-bab0-74e46a86b1d9-kube-api-access-tj8pd\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:32.042396 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:32.042368    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:32.048831 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:32.048802    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-serving-fail-predictor-5db8cf774f-6td4m"]
Oct 31 01:22:33.199063 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:33.199023    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" path="/var/lib/kubelet/pods/0b66f70e-6199-4322-bab0-74e46a86b1d9/volumes"
Oct 31 01:22:34.428555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.428523    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.428970    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.428990    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429006    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429012    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429045    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="storage-initializer"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429052    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="storage-initializer"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429065    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kube-rbac-proxy"
Oct 31 01:22:34.429101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429072    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kube-rbac-proxy"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429146    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kube-rbac-proxy"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429157    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429168    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b90f612-d2ac-457a-b7a5-040df3902b50" containerName="kserve-container"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429178    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429264    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.429519 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.429273    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b66f70e-6199-4322-bab0-74e46a86b1d9" containerName="storage-initializer"
Oct 31 01:22:34.433934 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.433913    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.439938 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.439908    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440438    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440470    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440473    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440475    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440528    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-predictor-serving-cert\""
Oct 31 01:22:34.440696 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.440564    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:22:34.445456 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.445433    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:22:34.447223 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.447197    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:34.573045 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.573008    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.573217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.573059    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.573217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.573076    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.573217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.573178    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.573344 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.573240    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5rq\" (UniqueName: \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.588722 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.588693    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:34.674652 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.674617    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.674861 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.674666    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.674861 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.674709    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.674861 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.674820    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.675031 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.674877    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5rq\" (UniqueName: \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.675093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.675073    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.675428 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.675405    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.675428 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.675418    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.677453 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.677436    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.691264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.691201    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5rq\" (UniqueName: \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\") pod \"isvc-paddle-predictor-db7449db4-grkzt\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") " pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:34.752130 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:34.752094    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:35.030640 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:35.030562    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:22:35.032895 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:22:35.032869    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc37b21_5e59_46d4_86f9_45005c27f548.slice/crio-c44b62e5bca24d9f87295c5f9282d589aa642d3188b2037ab8e56d681ac41791 WatchSource:0}: Error finding container c44b62e5bca24d9f87295c5f9282d589aa642d3188b2037ab8e56d681ac41791: Status 404 returned error can't find the container with id c44b62e5bca24d9f87295c5f9282d589aa642d3188b2037ab8e56d681ac41791
Oct 31 01:22:35.198819 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:35.198787    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:22:35.716547 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:35.716512    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerStarted","Data":"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"}
Oct 31 01:22:35.716959 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:35.716570    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerStarted","Data":"c44b62e5bca24d9f87295c5f9282d589aa642d3188b2037ab8e56d681ac41791"}
Oct 31 01:22:40.738173 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:40.738138    4553 generic.go:356] "Generic (PLEG): container finished" podID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerID="f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840" exitCode=0
Oct 31 01:22:40.738566 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:40.738210    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerDied","Data":"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"}
Oct 31 01:22:44.256251 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.256117    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:22:44.257503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.257101    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" containerID="cri-o://db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc" gracePeriod=30
Oct 31 01:22:44.257503 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.257390    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kube-rbac-proxy" containerID="cri-o://c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068" gracePeriod=30
Oct 31 01:22:44.330833 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.330795    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:44.762005 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.761970    4553 generic.go:356] "Generic (PLEG): container finished" podID="77a194fe-cef4-4530-af2e-aaba29300035" containerID="c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068" exitCode=0
Oct 31 01:22:44.762201 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:44.762014    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerDied","Data":"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"}
Oct 31 01:22:45.196028 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:45.195980    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 31 01:22:47.331923 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:47.331879    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Oct 31 01:22:48.011162 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:48.011125    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:51.228476 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.228452    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:22:51.320128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320095    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\") pod \"77a194fe-cef4-4530-af2e-aaba29300035\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") "
Oct 31 01:22:51.320128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320134    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\") pod \"77a194fe-cef4-4530-af2e-aaba29300035\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") "
Oct 31 01:22:51.320397 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320185    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\" (UniqueName: \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\") pod \"77a194fe-cef4-4530-af2e-aaba29300035\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") "
Oct 31 01:22:51.320397 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320212    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\") pod \"77a194fe-cef4-4530-af2e-aaba29300035\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") "
Oct 31 01:22:51.320397 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320244    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\") pod \"77a194fe-cef4-4530-af2e-aaba29300035\" (UID: \"77a194fe-cef4-4530-af2e-aaba29300035\") "
Oct 31 01:22:51.320848 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320823    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config" pod "77a194fe-cef4-4530-af2e-aaba29300035" (UID: "77a194fe-cef4-4530-af2e-aaba29300035"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:51.320955 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.320857    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location" pod "77a194fe-cef4-4530-af2e-aaba29300035" (UID: "77a194fe-cef4-4530-af2e-aaba29300035"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:22:51.321055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.321023    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert" pod "77a194fe-cef4-4530-af2e-aaba29300035" (UID: "77a194fe-cef4-4530-af2e-aaba29300035"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:51.322883 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.322844    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls" pod "77a194fe-cef4-4530-af2e-aaba29300035" (UID: "77a194fe-cef4-4530-af2e-aaba29300035"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:51.323318 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.323293    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948" pod "77a194fe-cef4-4530-af2e-aaba29300035" (UID: "77a194fe-cef4-4530-af2e-aaba29300035"). InnerVolumeSpecName "kube-api-access-nf948". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:51.421413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.421373    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf948\" (UniqueName: \"kubernetes.io/projected/77a194fe-cef4-4530-af2e-aaba29300035-kube-api-access-nf948\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:51.421413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.421411    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:51.421413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.421422    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a194fe-cef4-4530-af2e-aaba29300035-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:51.421659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.421431    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a194fe-cef4-4530-af2e-aaba29300035-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:51.421659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.421440    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a194fe-cef4-4530-af2e-aaba29300035-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:22:51.795146 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.795062    4553 generic.go:356] "Generic (PLEG): container finished" podID="77a194fe-cef4-4530-af2e-aaba29300035" containerID="db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc" exitCode=0
Oct 31 01:22:51.795146 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.795118    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerDied","Data":"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"}
Oct 31 01:22:51.795338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.795150    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"
Oct 31 01:22:51.795338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.795177    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v" event={"ID":"77a194fe-cef4-4530-af2e-aaba29300035","Type":"ContainerDied","Data":"bf7e1adce2a99d9f24e500f41f930aa58e51825492d96520884e030e088acb64"}
Oct 31 01:22:51.795338 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.795205    4553 scope.go:122] "RemoveContainer" containerID="c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"
Oct 31 01:22:51.807376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.807348    4553 scope.go:122] "RemoveContainer" containerID="db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"
Oct 31 01:22:51.817143 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.817068    4553 scope.go:122] "RemoveContainer" containerID="f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"
Oct 31 01:22:51.826436 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.826415    4553 scope.go:122] "RemoveContainer" containerID="c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"
Oct 31 01:22:51.826771 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:51.826741    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068\": container with ID starting with c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068 not found: ID does not exist" containerID="c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"
Oct 31 01:22:51.826856 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.826769    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068"} err="failed to get container status \"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068\": rpc error: code = NotFound desc = could not find container \"c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068\": container with ID starting with c496676ea6407467ff278eb20a9a8da5e748755d2b4ac575ab6ed2eaf441c068 not found: ID does not exist"
Oct 31 01:22:51.826856 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.826796    4553 scope.go:122] "RemoveContainer" containerID="db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"
Oct 31 01:22:51.827083 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:51.827059    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc\": container with ID starting with db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc not found: ID does not exist" containerID="db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"
Oct 31 01:22:51.827132 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.827091    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc"} err="failed to get container status \"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc\": rpc error: code = NotFound desc = could not find container \"db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc\": container with ID starting with db8627769b4ef6f42797ec77902f95ff5ecbdcafc16a519e5fba31c59175b9dc not found: ID does not exist"
Oct 31 01:22:51.827132 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.827111    4553 scope.go:122] "RemoveContainer" containerID="f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"
Oct 31 01:22:51.827422 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:51.827386    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453\": container with ID starting with f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453 not found: ID does not exist" containerID="f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"
Oct 31 01:22:51.827509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.827418    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453"} err="failed to get container status \"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453\": rpc error: code = NotFound desc = could not find container \"f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453\": container with ID starting with f89050778711e22bb9268bc69679b5aaedc0e2f64ad81d02bd3f08d268206453 not found: ID does not exist"
Oct 31 01:22:51.830192 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.830171    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:22:51.836541 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:51.836521    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-predictor-85d97887df-kgb7v"]
Oct 31 01:22:52.801500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.801465    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerStarted","Data":"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"}
Oct 31 01:22:52.801979 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.801564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerStarted","Data":"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"}
Oct 31 01:22:52.801979 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.801848    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:52.802094 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.802037    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:52.803301 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.803276    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:22:52.831611 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:52.831565    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podStartSLOduration=2.188894384 podStartE2EDuration="18.83155096s" totalImagesPullingTime="11.609170669s" totalInitContainerRuntime="5.033485907s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:52.828927586 +0000 UTC m=+3235.737930253" watchObservedRunningTime="2026-10-31 01:22:52.83155096 +0000 UTC m=+3235.740553614"
Oct 31 01:22:53.198939 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:53.198900    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a194fe-cef4-4530-af2e-aaba29300035" path="/var/lib/kubelet/pods/77a194fe-cef4-4530-af2e-aaba29300035/volumes"
Oct 31 01:22:53.745058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:53.745020    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:53.806895 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:53.806860    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:22:53.885555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:53.885514    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:57.128782 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:22:57.128747    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:22:58.811099 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:58.811066    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:22:58.811572 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:22:58.811547    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:23:08.812483 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:08.812438    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:23:18.812426 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:18.812385    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:23:27.129975 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:23:27.129931    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:23:28.812185 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:28.812152    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:23:38.812447 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:38.812360    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:23:46.514907 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:46.514861    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:23:46.515508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:46.515124    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" containerID="cri-o://685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9" gracePeriod=30
Oct 31 01:23:46.515508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:46.515164    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kube-rbac-proxy" containerID="cri-o://596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff" gracePeriod=30
Oct 31 01:23:47.003928 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:47.003897    4553 generic.go:356] "Generic (PLEG): container finished" podID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerID="596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff" exitCode=0
Oct 31 01:23:47.004100 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:47.003952    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerDied","Data":"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"}
Oct 31 01:23:48.112845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:48.112811    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:48.251863 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:48.251827    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:48.807998 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:48.807958    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 31 01:23:48.812333 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:48.812302    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 31 01:23:49.582654 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.582631    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:23:49.639922 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.639846    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\" (UniqueName: \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\") pod \"5cc37b21-5e59-46d4-86f9-45005c27f548\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") "
Oct 31 01:23:49.640074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.639924    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\") pod \"5cc37b21-5e59-46d4-86f9-45005c27f548\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") "
Oct 31 01:23:49.640074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.639964    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\") pod \"5cc37b21-5e59-46d4-86f9-45005c27f548\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") "
Oct 31 01:23:49.640074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.639987    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\") pod \"5cc37b21-5e59-46d4-86f9-45005c27f548\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") "
Oct 31 01:23:49.640074 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.640009    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\") pod \"5cc37b21-5e59-46d4-86f9-45005c27f548\" (UID: \"5cc37b21-5e59-46d4-86f9-45005c27f548\") "
Oct 31 01:23:49.640325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.640303    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert" pod "5cc37b21-5e59-46d4-86f9-45005c27f548" (UID: "5cc37b21-5e59-46d4-86f9-45005c27f548"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:49.640390 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.640344    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config" pod "5cc37b21-5e59-46d4-86f9-45005c27f548" (UID: "5cc37b21-5e59-46d4-86f9-45005c27f548"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:49.642312 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.642288    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls" pod "5cc37b21-5e59-46d4-86f9-45005c27f548" (UID: "5cc37b21-5e59-46d4-86f9-45005c27f548"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:49.642434 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.642320    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq" pod "5cc37b21-5e59-46d4-86f9-45005c27f548" (UID: "5cc37b21-5e59-46d4-86f9-45005c27f548"). InnerVolumeSpecName "kube-api-access-jh5rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:49.648586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.648558    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location" pod "5cc37b21-5e59-46d4-86f9-45005c27f548" (UID: "5cc37b21-5e59-46d4-86f9-45005c27f548"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:49.740504 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.740457    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:49.740504 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.740500    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc37b21-5e59-46d4-86f9-45005c27f548-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:49.740504 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.740511    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc37b21-5e59-46d4-86f9-45005c27f548-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:49.740504 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.740520    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:49.740804 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:49.740530    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5rq\" (UniqueName: \"kubernetes.io/projected/5cc37b21-5e59-46d4-86f9-45005c27f548-kube-api-access-jh5rq\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.016311 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.016271    4553 generic.go:356] "Generic (PLEG): container finished" podID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerID="685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9" exitCode=0
Oct 31 01:23:50.016480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.016328    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerDied","Data":"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"}
Oct 31 01:23:50.016480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.016363    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"
Oct 31 01:23:50.016480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.016420    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt" event={"ID":"5cc37b21-5e59-46d4-86f9-45005c27f548","Type":"ContainerDied","Data":"c44b62e5bca24d9f87295c5f9282d589aa642d3188b2037ab8e56d681ac41791"}
Oct 31 01:23:50.016480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.016458    4553 scope.go:122] "RemoveContainer" containerID="596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"
Oct 31 01:23:50.025152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.025135    4553 scope.go:122] "RemoveContainer" containerID="685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"
Oct 31 01:23:50.032559 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.032537    4553 scope.go:122] "RemoveContainer" containerID="f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"
Oct 31 01:23:50.039299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.039281    4553 scope.go:122] "RemoveContainer" containerID="596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"
Oct 31 01:23:50.039559 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:23:50.039538    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff\": container with ID starting with 596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff not found: ID does not exist" containerID="596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"
Oct 31 01:23:50.039622 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.039568    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff"} err="failed to get container status \"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff\": rpc error: code = NotFound desc = could not find container \"596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff\": container with ID starting with 596c53cdf452eb20d38992834484f914d03ea391bbf80df9d8d4f96209c12eff not found: ID does not exist"
Oct 31 01:23:50.039622 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.039584    4553 scope.go:122] "RemoveContainer" containerID="685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"
Oct 31 01:23:50.039857 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:23:50.039842    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9\": container with ID starting with 685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9 not found: ID does not exist" containerID="685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"
Oct 31 01:23:50.039909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.039861    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9"} err="failed to get container status \"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9\": rpc error: code = NotFound desc = could not find container \"685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9\": container with ID starting with 685f29dc4f917e0a5256039c3cb7f12f6ba53408d7e510aa0e8eca308c3418d9 not found: ID does not exist"
Oct 31 01:23:50.039909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.039876    4553 scope.go:122] "RemoveContainer" containerID="f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"
Oct 31 01:23:50.040103 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:23:50.040087    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840\": container with ID starting with f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840 not found: ID does not exist" containerID="f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"
Oct 31 01:23:50.040139 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.040108    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840"} err="failed to get container status \"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840\": rpc error: code = NotFound desc = could not find container \"f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840\": container with ID starting with f963d6462c851960c290cb449d9bb2f48eae43a3fc34b489d6588de6e6b65840 not found: ID does not exist"
Oct 31 01:23:50.042224 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.042205    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:23:50.051262 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:50.051241    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-predictor-db7449db4-grkzt"]
Oct 31 01:23:51.199103 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.199068    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" path="/var/lib/kubelet/pods/5cc37b21-5e59-46d4-86f9-45005c27f548/volumes"
Oct 31 01:23:51.951969 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.951930    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:51.959937 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.959912    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:23:51.960217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960205    4553 cpu_manager.go:414] "containerMap: removing container" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960220    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960230    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960237    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960253    4553 cpu_manager.go:414] "containerMap: removing container" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container"
Oct 31 01:23:51.960260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960259    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960273    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="storage-initializer"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960278    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="storage-initializer"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960286    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960290    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960298    4553 cpu_manager.go:414] "containerMap: removing container" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="storage-initializer"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960302    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="storage-initializer"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960352    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kserve-container"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960360    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960366    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc37b21-5e59-46d4-86f9-45005c27f548" containerName="kube-rbac-proxy"
Oct 31 01:23:51.960440 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.960374    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a194fe-cef4-4530-af2e-aaba29300035" containerName="kserve-container"
Oct 31 01:23:51.964752 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.964737    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:51.971475 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.971453    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Oct 31 01:23:51.971595 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.971559    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:23:51.972018 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.972000    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Oct 31 01:23:51.972101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.972028    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:23:51.972101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.972054    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:23:51.972101 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.972072    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:23:51.972218 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.972164    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:23:51.977559 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:51.977538    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:23:52.056596 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.056562    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.056596 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.056597    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcg5\" (UniqueName: \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.056832 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.056630    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.056832 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.056773    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.056899 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.056832    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.087769 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.087718    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:52.158010 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.157981    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158147 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158035    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158147 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158103    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158147 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158120    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcg5\" (UniqueName: \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158147 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158145    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158493 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158468    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158782 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158763    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.158851 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.158802    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.160837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.160815    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.173270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.173247    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcg5\" (UniqueName: \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\") pod \"isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") " pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.283351 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.283286    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:52.560362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:52.560224    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:23:52.562905 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:23:52.562875    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997def99_d06d_487e_b7d9_e6d341478c1e.slice/crio-39c42f37ff4afc599189bcaa334265df741df06ed91ebcd118f6d113ecb96ff3 WatchSource:0}: Error finding container 39c42f37ff4afc599189bcaa334265df741df06ed91ebcd118f6d113ecb96ff3: Status 404 returned error can't find the container with id 39c42f37ff4afc599189bcaa334265df741df06ed91ebcd118f6d113ecb96ff3
Oct 31 01:23:53.030321 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:53.030283    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerStarted","Data":"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"}
Oct 31 01:23:53.030509 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:53.030345    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerStarted","Data":"39c42f37ff4afc599189bcaa334265df741df06ed91ebcd118f6d113ecb96ff3"}
Oct 31 01:23:57.136574 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:23:57.136532    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:23:57.447174 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:57.447147    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:23:57.453890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:57.453869    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:23:58.051143 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:58.051109    4553 generic.go:356] "Generic (PLEG): container finished" podID="997def99-d06d-487e-b7d9-e6d341478c1e" containerID="2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3" exitCode=0
Oct 31 01:23:58.051299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:58.051182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerDied","Data":"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"}
Oct 31 01:23:59.057479 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:59.057442    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerStarted","Data":"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"}
Oct 31 01:23:59.057905 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:59.057513    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerStarted","Data":"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"}
Oct 31 01:23:59.057905 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:59.057781    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:23:59.086144 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:23:59.086093    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podStartSLOduration=2.7492635549999997 podStartE2EDuration="8.086080227s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.336816672s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:59.085304449 +0000 UTC m=+3301.994307095" watchObservedRunningTime="2026-10-31 01:23:59.086080227 +0000 UTC m=+3301.995082881"
Oct 31 01:24:00.061504 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:00.061474    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:24:00.063185 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:00.063145    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:01.065640 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:01.065594    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:06.070395 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:06.070365    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:24:06.070904 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:06.070878    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:16.071481 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:16.071437    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:22.844799 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:22.844763    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:26.070845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:26.070800    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:27.130448 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:24:27.130405    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:24:36.071806 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:36.071755    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:45.497170 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:45.497135    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:46.071656 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:46.071624    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:24:48.121546 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:48.121507    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:52.886167 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:52.886128    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:53.031434 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:53.031397    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:53.662565 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:53.662534    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:24:53.662860 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:53.662792    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" containerID="cri-o://87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214" gracePeriod=30
Oct 31 01:24:53.662930 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:53.662881    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kube-rbac-proxy" containerID="cri-o://883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3" gracePeriod=30
Oct 31 01:24:53.734963 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:53.734932    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:54.251734 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:54.251700    4553 generic.go:356] "Generic (PLEG): container finished" podID="997def99-d06d-487e-b7d9-e6d341478c1e" containerID="883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3" exitCode=0
Oct 31 01:24:54.252137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:54.251744    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerDied","Data":"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"}
Oct 31 01:24:56.066159 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.066115    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Oct 31 01:24:56.071470 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.071441    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 31 01:24:56.504978 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.504954    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:24:56.554221 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554194    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\") pod \"997def99-d06d-487e-b7d9-e6d341478c1e\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") "
Oct 31 01:24:56.554330 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554237    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\") pod \"997def99-d06d-487e-b7d9-e6d341478c1e\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") "
Oct 31 01:24:56.554330 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554257    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\" (UniqueName: \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\") pod \"997def99-d06d-487e-b7d9-e6d341478c1e\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") "
Oct 31 01:24:56.554330 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554295    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\") pod \"997def99-d06d-487e-b7d9-e6d341478c1e\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") "
Oct 31 01:24:56.554460 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554349    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\") pod \"997def99-d06d-487e-b7d9-e6d341478c1e\" (UID: \"997def99-d06d-487e-b7d9-e6d341478c1e\") "
Oct 31 01:24:56.554578 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554553    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert" pod "997def99-d06d-487e-b7d9-e6d341478c1e" (UID: "997def99-d06d-487e-b7d9-e6d341478c1e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:56.554648 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.554599    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config" pod "997def99-d06d-487e-b7d9-e6d341478c1e" (UID: "997def99-d06d-487e-b7d9-e6d341478c1e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:56.556569 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.556543    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls" pod "997def99-d06d-487e-b7d9-e6d341478c1e" (UID: "997def99-d06d-487e-b7d9-e6d341478c1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:56.556569 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.556559    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5" pod "997def99-d06d-487e-b7d9-e6d341478c1e" (UID: "997def99-d06d-487e-b7d9-e6d341478c1e"). InnerVolumeSpecName "kube-api-access-7qcg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:56.563760 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.563737    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location" pod "997def99-d06d-487e-b7d9-e6d341478c1e" (UID: "997def99-d06d-487e-b7d9-e6d341478c1e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:56.655450 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.655422    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:56.655542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.655449    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997def99-d06d-487e-b7d9-e6d341478c1e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:56.655542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.655464    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qcg5\" (UniqueName: \"kubernetes.io/projected/997def99-d06d-487e-b7d9-e6d341478c1e-kube-api-access-7qcg5\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:56.655542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.655477    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997def99-d06d-487e-b7d9-e6d341478c1e-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:56.655542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:56.655490    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997def99-d06d-487e-b7d9-e6d341478c1e-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:57.129731 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:24:57.129635    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:24:57.264863 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.264830    4553 generic.go:356] "Generic (PLEG): container finished" podID="997def99-d06d-487e-b7d9-e6d341478c1e" containerID="87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214" exitCode=0
Oct 31 01:24:57.265022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.264876    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerDied","Data":"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"}
Oct 31 01:24:57.265022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.264893    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"
Oct 31 01:24:57.265022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.264921    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh" event={"ID":"997def99-d06d-487e-b7d9-e6d341478c1e","Type":"ContainerDied","Data":"39c42f37ff4afc599189bcaa334265df741df06ed91ebcd118f6d113ecb96ff3"}
Oct 31 01:24:57.265022 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.264946    4553 scope.go:122] "RemoveContainer" containerID="883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"
Oct 31 01:24:57.273068 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.273045    4553 scope.go:122] "RemoveContainer" containerID="87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"
Oct 31 01:24:57.280802 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.280782    4553 scope.go:122] "RemoveContainer" containerID="2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"
Oct 31 01:24:57.287650 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.287633    4553 scope.go:122] "RemoveContainer" containerID="883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"
Oct 31 01:24:57.287994 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:24:57.287969    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3\": container with ID starting with 883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3 not found: ID does not exist" containerID="883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"
Oct 31 01:24:57.288122 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.288004    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3"} err="failed to get container status \"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3\": rpc error: code = NotFound desc = could not find container \"883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3\": container with ID starting with 883343225a53d14fa2fc5ddc49c7c19b94bb74093110713634d32662809279f3 not found: ID does not exist"
Oct 31 01:24:57.288122 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.288025    4553 scope.go:122] "RemoveContainer" containerID="87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"
Oct 31 01:24:57.288805 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:24:57.288776    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214\": container with ID starting with 87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214 not found: ID does not exist" containerID="87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"
Oct 31 01:24:57.288896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.288814    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214"} err="failed to get container status \"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214\": rpc error: code = NotFound desc = could not find container \"87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214\": container with ID starting with 87b65659ff370e72a1dc9beb62f236d5a5624f1b40eeacb614770b8ebff99214 not found: ID does not exist"
Oct 31 01:24:57.288896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.288838    4553 scope.go:122] "RemoveContainer" containerID="2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"
Oct 31 01:24:57.289152 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:24:57.289119    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3\": container with ID starting with 2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3 not found: ID does not exist" containerID="2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"
Oct 31 01:24:57.289217 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.289151    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3"} err="failed to get container status \"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3\": rpc error: code = NotFound desc = could not find container \"2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3\": container with ID starting with 2cc5098cf8c9386fe6c3c465dd23b33cb44fdfb552abb91aab05efca007a0ef3 not found: ID does not exist"
Oct 31 01:24:57.289659 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.289635    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:24:57.295640 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:57.295621    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-runtime-predictor-64fcbcfc4d-kr9dh"]
Oct 31 01:24:59.122070 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122039    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122348    4553 cpu_manager.go:414] "containerMap: removing container" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122358    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122368    4553 cpu_manager.go:414] "containerMap: removing container" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kube-rbac-proxy"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122373    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kube-rbac-proxy"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122381    4553 cpu_manager.go:414] "containerMap: removing container" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="storage-initializer"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122387    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="storage-initializer"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122441    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kserve-container"
Oct 31 01:24:59.122467 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.122447    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" containerName="kube-rbac-proxy"
Oct 31 01:24:59.126817 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.126787    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:59.126950 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.126826    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.132890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.132864    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:24:59.132890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.132879    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 31 01:24:59.132890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.132888    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:24:59.132890 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.132875    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:24:59.133571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.133551    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Oct 31 01:24:59.133757 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.133742    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:24:59.133916 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.133773    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:24:59.137623 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.137600    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:24:59.199596 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.199571    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997def99-d06d-487e-b7d9-e6d341478c1e" path="/var/lib/kubelet/pods/997def99-d06d-487e-b7d9-e6d341478c1e/volumes"
Oct 31 01:24:59.270979 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.270947    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:59.273784 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.273764    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.273902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.273791    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.273902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.273819    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsz9\" (UniqueName: \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.273902 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.273843    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.274063 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.274014    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.374941 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.374874    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.374941 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.374913    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375115 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.374959    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsz9\" (UniqueName: \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375115 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.374986    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375115 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.375026    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375422 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.375403    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.375501    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.375613 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.375597    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.377608 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.377591    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.389998 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.389977    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsz9\" (UniqueName: \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\") pod \"isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") " pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.445927 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.445897    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:24:59.721766 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:24:59.721740    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:24:59.723956 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:24:59.723928    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f708d_ac27_4ce4_aa0c_8fa83c316337.slice/crio-e3f3c16837e0bd5053e6108265743fbbabf64c4fa3141a302402527a81fda920 WatchSource:0}: Error finding container e3f3c16837e0bd5053e6108265743fbbabf64c4fa3141a302402527a81fda920: Status 404 returned error can't find the container with id e3f3c16837e0bd5053e6108265743fbbabf64c4fa3141a302402527a81fda920
Oct 31 01:25:00.284206 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:00.284168    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerStarted","Data":"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"}
Oct 31 01:25:00.284746 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:00.284258    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerStarted","Data":"e3f3c16837e0bd5053e6108265743fbbabf64c4fa3141a302402527a81fda920"}
Oct 31 01:25:05.302425 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:05.302389    4553 generic.go:356] "Generic (PLEG): container finished" podID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerID="19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7" exitCode=0
Oct 31 01:25:05.302425 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:05.302429    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerDied","Data":"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"}
Oct 31 01:25:06.307798 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.307761    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerStarted","Data":"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"}
Oct 31 01:25:06.308219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.307826    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerStarted","Data":"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"}
Oct 31 01:25:06.308219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.308132    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:25:06.308219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.308178    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:25:06.309548 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.309519    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:06.335658 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:06.335614    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podStartSLOduration=2.419566034 podStartE2EDuration="7.33560445s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.916038416s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:06.334519641 +0000 UTC m=+3369.243522321" watchObservedRunningTime="2026-10-31 01:25:06.33560445 +0000 UTC m=+3369.244607104"
Oct 31 01:25:07.311070 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:07.311033    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:12.316191 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:12.316109    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:25:12.316847 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:12.316815    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:22.317552 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:22.317505    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:27.130081 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:25:27.130036    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:25:32.317325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:32.317284    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:42.317128 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:42.317087    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:25:43.265280 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:43.265236    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.610484 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.610450    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:25:46.614030 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.614013    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.621284 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.621266    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\""
Oct 31 01:25:46.621413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.621268    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 31 01:25:46.621413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.621291    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\""
Oct 31 01:25:46.621413 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.621322    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:25:46.621528 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.621451    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\""
Oct 31 01:25:46.622309 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.622286    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Oct 31 01:25:46.622416 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.622309    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\""
Oct 31 01:25:46.623071 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.623050    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.626112 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.626090    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:25:46.638303 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.638273    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.638401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.638322    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htl85\" (UniqueName: \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.638401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.638339    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.638522 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.638492    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.638617 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.638590    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.739533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.739489    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.739533 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.739535    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.739750 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.739588    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htl85\" (UniqueName: \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.739750 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.739611    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.739750 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.739648    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.740055 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.740036    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.740365 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.740347    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.740424 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.740347    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.742126 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.742108    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.754478 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.754453    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htl85\" (UniqueName: \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\") pod \"isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:46.770152 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.770125    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.931500 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:46.931468    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:47.217279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:47.217135    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:25:47.220199 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:25:47.220167    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cf71be_766c_44e7_891d_183fa9256fa6.slice/crio-d4d5727e2d88c597032de080c6075672adbd27ed3d9ce950907d0bc7f863e622 WatchSource:0}: Error finding container d4d5727e2d88c597032de080c6075672adbd27ed3d9ce950907d0bc7f863e622: Status 404 returned error can't find the container with id d4d5727e2d88c597032de080c6075672adbd27ed3d9ce950907d0bc7f863e622
Oct 31 01:25:47.452171 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:47.452133    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerStarted","Data":"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"}
Oct 31 01:25:47.452346 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:47.452214    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerStarted","Data":"d4d5727e2d88c597032de080c6075672adbd27ed3d9ce950907d0bc7f863e622"}
Oct 31 01:25:48.175459 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:48.175423    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:51.474032 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:51.473998    4553 generic.go:356] "Generic (PLEG): container finished" podID="15cf71be-766c-44e7-891d-183fa9256fa6" containerID="5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e" exitCode=0
Oct 31 01:25:51.474478 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:51.474068    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerDied","Data":"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"}
Oct 31 01:25:52.317360 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.317322    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:25:52.480806 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.480770    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerStarted","Data":"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"}
Oct 31 01:25:52.481219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.480833    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerStarted","Data":"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"}
Oct 31 01:25:52.481219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.481134    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:52.481348 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.481297    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:52.482859 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.482834    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Oct 31 01:25:52.506463 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:52.506416    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podStartSLOduration=2.745649049 podStartE2EDuration="6.506406308s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760757259s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:52.505663855 +0000 UTC m=+3415.414666535" watchObservedRunningTime="2026-10-31 01:25:52.506406308 +0000 UTC m=+3415.415408991"
Oct 31 01:25:53.484964 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:53.484930    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Oct 31 01:25:57.130185 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:25:57.130145    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:25:58.489188 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:58.489161    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:25:58.489666 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:25:58.489628    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Oct 31 01:26:01.074399 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.074368    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:26:01.074899 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.074697    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" containerID="cri-o://26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330" gracePeriod=30
Oct 31 01:26:01.074899 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.074757    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kube-rbac-proxy" containerID="cri-o://6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27" gracePeriod=30
Oct 31 01:26:01.146046 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.146014    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:01.517270 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.517232    4553 generic.go:356] "Generic (PLEG): container finished" podID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerID="6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27" exitCode=0
Oct 31 01:26:01.517445 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:01.517302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerDied","Data":"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"}
Oct 31 01:26:02.311837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:02.311789    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Oct 31 01:26:02.317180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:02.317140    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 31 01:26:03.914108 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.914080    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:26:03.986277 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986250    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\") pod \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") "
Oct 31 01:26:03.986433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986298    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\" (UniqueName: \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\") pod \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") "
Oct 31 01:26:03.986433 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986328    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\") pod \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") "
Oct 31 01:26:03.986529 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986440    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\") pod \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") "
Oct 31 01:26:03.986587 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986538    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\") pod \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\" (UID: \"0d2f708d-ac27-4ce4-aa0c-8fa83c316337\") "
Oct 31 01:26:03.986650 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986632    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert" pod "0d2f708d-ac27-4ce4-aa0c-8fa83c316337" (UID: "0d2f708d-ac27-4ce4-aa0c-8fa83c316337"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:03.986851 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986826    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config" pod "0d2f708d-ac27-4ce4-aa0c-8fa83c316337" (UID: "0d2f708d-ac27-4ce4-aa0c-8fa83c316337"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:03.986959 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986940    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:03.987014 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.986968    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:03.988569 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.988541    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls" pod "0d2f708d-ac27-4ce4-aa0c-8fa83c316337" (UID: "0d2f708d-ac27-4ce4-aa0c-8fa83c316337"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:03.988689 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.988591    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9" pod "0d2f708d-ac27-4ce4-aa0c-8fa83c316337" (UID: "0d2f708d-ac27-4ce4-aa0c-8fa83c316337"). InnerVolumeSpecName "kube-api-access-zzsz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:03.995901 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:03.995871    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location" pod "0d2f708d-ac27-4ce4-aa0c-8fa83c316337" (UID: "0d2f708d-ac27-4ce4-aa0c-8fa83c316337"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:04.088260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.088194    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:04.088260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.088222    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:04.088260 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.088232    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzsz9\" (UniqueName: \"kubernetes.io/projected/0d2f708d-ac27-4ce4-aa0c-8fa83c316337-kube-api-access-zzsz9\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:04.530184 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.530154    4553 generic.go:356] "Generic (PLEG): container finished" podID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerID="26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330" exitCode=0
Oct 31 01:26:04.530350 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.530214    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"
Oct 31 01:26:04.530350 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.530232    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerDied","Data":"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"}
Oct 31 01:26:04.530350 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.530291    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq" event={"ID":"0d2f708d-ac27-4ce4-aa0c-8fa83c316337","Type":"ContainerDied","Data":"e3f3c16837e0bd5053e6108265743fbbabf64c4fa3141a302402527a81fda920"}
Oct 31 01:26:04.530350 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.530321    4553 scope.go:122] "RemoveContainer" containerID="6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"
Oct 31 01:26:04.539180 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.539160    4553 scope.go:122] "RemoveContainer" containerID="26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"
Oct 31 01:26:04.546996 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.546978    4553 scope.go:122] "RemoveContainer" containerID="19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"
Oct 31 01:26:04.554193 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.554175    4553 scope.go:122] "RemoveContainer" containerID="6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"
Oct 31 01:26:04.554457 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:04.554436    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27\": container with ID starting with 6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27 not found: ID does not exist" containerID="6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"
Oct 31 01:26:04.554508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.554468    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27"} err="failed to get container status \"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27\": rpc error: code = NotFound desc = could not find container \"6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27\": container with ID starting with 6f32833872d32033ebd33448efaad18218f42cf64a83c38168527d6283bd1f27 not found: ID does not exist"
Oct 31 01:26:04.554508 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.554494    4553 scope.go:122] "RemoveContainer" containerID="26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"
Oct 31 01:26:04.555018 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:04.554888    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330\": container with ID starting with 26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330 not found: ID does not exist" containerID="26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"
Oct 31 01:26:04.555018 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.554920    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330"} err="failed to get container status \"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330\": rpc error: code = NotFound desc = could not find container \"26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330\": container with ID starting with 26fcfd34f0e44af8ab91d2de3fc12a72cc10f8aa15e757a6c75033217e426330 not found: ID does not exist"
Oct 31 01:26:04.555018 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.554941    4553 scope.go:122] "RemoveContainer" containerID="19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"
Oct 31 01:26:04.555352 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:04.555319    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7\": container with ID starting with 19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7 not found: ID does not exist" containerID="19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"
Oct 31 01:26:04.555352 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.555345    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7"} err="failed to get container status \"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7\": rpc error: code = NotFound desc = could not find container \"19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7\": container with ID starting with 19860b7a990307e7f22d94d6fdd28b8c023fb8927f4d8e92f9661b5c68bc52b7 not found: ID does not exist"
Oct 31 01:26:04.556706 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.556667    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:26:04.564695 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:04.564653    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-paddle-v2-kserve-predictor-5ddcb79644-7lbsq"]
Oct 31 01:26:05.199929 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:05.199899    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" path="/var/lib/kubelet/pods/0d2f708d-ac27-4ce4-aa0c-8fa83c316337/volumes"
Oct 31 01:26:06.527521 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527487    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527811    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527823    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527831    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kube-rbac-proxy"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527836    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kube-rbac-proxy"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527846    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="storage-initializer"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527851    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="storage-initializer"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527897    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kserve-container"
Oct 31 01:26:06.528023 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.527905    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f708d-ac27-4ce4-aa0c-8fa83c316337" containerName="kube-rbac-proxy"
Oct 31 01:26:06.530710 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.530665    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:06.532099 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.532079    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.540824 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.540804    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:26:06.541376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541312    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:26:06.541376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541342    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:26:06.541376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541353    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-predictor-serving-cert\""
Oct 31 01:26:06.541376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541361    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:26:06.541376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541358    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:26:06.541594 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.541380    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Oct 31 01:26:06.543983 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.543953    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:26:06.607835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.607806    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.607977 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.607843    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.607977 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.607877    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6gp\" (UniqueName: \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.607977 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.607947    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.608094 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.607978    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.666888 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.666859    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:06.709039 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709014    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709048    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709070    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6gp\" (UniqueName: \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709109    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709155 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709136    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709545 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709523    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709796 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709778    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.709854 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.709831    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.711669 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.711648    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.723823 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.723803    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6gp\" (UniqueName: \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\") pod \"isvc-pmml-predictor-86668677f7-4zldc\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") " pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:06.849937 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:06.849861    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:07.127220 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:07.127194    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:26:07.129900 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:26:07.129876    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a7ef60_b6c6_42c4_a723_23eee6c32a41.slice/crio-977ceafb7264a8c02875f9e159b311ac06250359d71bd511a85808dc44a5285b WatchSource:0}: Error finding container 977ceafb7264a8c02875f9e159b311ac06250359d71bd511a85808dc44a5285b: Status 404 returned error can't find the container with id 977ceafb7264a8c02875f9e159b311ac06250359d71bd511a85808dc44a5285b
Oct 31 01:26:07.544695 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:07.544645    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerStarted","Data":"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"}
Oct 31 01:26:07.545093 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:07.544713    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerStarted","Data":"977ceafb7264a8c02875f9e159b311ac06250359d71bd511a85808dc44a5285b"}
Oct 31 01:26:08.490884 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:08.490850    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:26:11.562480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:11.562447    4553 generic.go:356] "Generic (PLEG): container finished" podID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerID="96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08" exitCode=0
Oct 31 01:26:11.562897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:11.562492    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerDied","Data":"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"}
Oct 31 01:26:11.563737 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:11.563720    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:26:16.993900 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:16.993870    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:26:16.994325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:16.994137    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container" containerID="cri-o://cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b" gracePeriod=30
Oct 31 01:26:16.994325 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:16.994201    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kube-rbac-proxy" containerID="cri-o://c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c" gracePeriod=30
Oct 31 01:26:17.074212 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:17.074164    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:17.609261 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:17.609227    4553 generic.go:356] "Generic (PLEG): container finished" podID="15cf71be-766c-44e7-891d-183fa9256fa6" containerID="c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c" exitCode=0
Oct 31 01:26:17.609436 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:17.609310    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerDied","Data":"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"}
Oct 31 01:26:17.860455 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:17.860397    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:26:18.004080 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004040    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\") pod \"15cf71be-766c-44e7-891d-183fa9256fa6\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") "
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004144    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\" (UniqueName: \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\") pod \"15cf71be-766c-44e7-891d-183fa9256fa6\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") "
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004185    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\") pod \"15cf71be-766c-44e7-891d-183fa9256fa6\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") "
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004217    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\") pod \"15cf71be-766c-44e7-891d-183fa9256fa6\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") "
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004428    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert" pod "15cf71be-766c-44e7-891d-183fa9256fa6" (UID: "15cf71be-766c-44e7-891d-183fa9256fa6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004491    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\") pod \"15cf71be-766c-44e7-891d-183fa9256fa6\" (UID: \"15cf71be-766c-44e7-891d-183fa9256fa6\") "
Oct 31 01:26:18.004555 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004549    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location" pod "15cf71be-766c-44e7-891d-183fa9256fa6" (UID: "15cf71be-766c-44e7-891d-183fa9256fa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:18.004872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004736    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15cf71be-766c-44e7-891d-183fa9256fa6-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:18.004872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004755    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:18.004872 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.004804    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config" pod "15cf71be-766c-44e7-891d-183fa9256fa6" (UID: "15cf71be-766c-44e7-891d-183fa9256fa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:18.006824 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.006788    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85" pod "15cf71be-766c-44e7-891d-183fa9256fa6" (UID: "15cf71be-766c-44e7-891d-183fa9256fa6"). InnerVolumeSpecName "kube-api-access-htl85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:18.007060 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.007028    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls" pod "15cf71be-766c-44e7-891d-183fa9256fa6" (UID: "15cf71be-766c-44e7-891d-183fa9256fa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:18.106114 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.106070    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htl85\" (UniqueName: \"kubernetes.io/projected/15cf71be-766c-44e7-891d-183fa9256fa6-kube-api-access-htl85\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:18.106114 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.106109    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15cf71be-766c-44e7-891d-183fa9256fa6-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:18.106335 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.106127    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15cf71be-766c-44e7-891d-183fa9256fa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:18.616298 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.616262    4553 generic.go:356] "Generic (PLEG): container finished" podID="15cf71be-766c-44e7-891d-183fa9256fa6" containerID="cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b" exitCode=0
Oct 31 01:26:18.616488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.616312    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerDied","Data":"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"}
Oct 31 01:26:18.616488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.616344    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"
Oct 31 01:26:18.616488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.616375    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w" event={"ID":"15cf71be-766c-44e7-891d-183fa9256fa6","Type":"ContainerDied","Data":"d4d5727e2d88c597032de080c6075672adbd27ed3d9ce950907d0bc7f863e622"}
Oct 31 01:26:18.616488 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.616412    4553 scope.go:122] "RemoveContainer" containerID="c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"
Oct 31 01:26:18.626953 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.626936    4553 scope.go:122] "RemoveContainer" containerID="cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"
Oct 31 01:26:18.635745 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.635724    4553 scope.go:122] "RemoveContainer" containerID="5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"
Oct 31 01:26:18.645204 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.645184    4553 scope.go:122] "RemoveContainer" containerID="c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"
Oct 31 01:26:18.645927 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:18.645905    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c\": container with ID starting with c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c not found: ID does not exist" containerID="c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"
Oct 31 01:26:18.646024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.645936    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c"} err="failed to get container status \"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c\": rpc error: code = NotFound desc = could not find container \"c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c\": container with ID starting with c884194790c7db5782271f427189ab7a609b240d53e93f6926b637f28ca9719c not found: ID does not exist"
Oct 31 01:26:18.646024 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.645952    4553 scope.go:122] "RemoveContainer" containerID="cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"
Oct 31 01:26:18.646261 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:18.646240    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b\": container with ID starting with cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b not found: ID does not exist" containerID="cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"
Oct 31 01:26:18.646355 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.646266    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b"} err="failed to get container status \"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b\": rpc error: code = NotFound desc = could not find container \"cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b\": container with ID starting with cc9e0d6390d826e4c0f7de5b20c572ccbf6500e52b5485f9df3948c6d9cb0b3b not found: ID does not exist"
Oct 31 01:26:18.646355 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.646280    4553 scope.go:122] "RemoveContainer" containerID="5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"
Oct 31 01:26:18.646611 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:18.646586    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e\": container with ID starting with 5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e not found: ID does not exist" containerID="5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"
Oct 31 01:26:18.646715 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.646617    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e"} err="failed to get container status \"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e\": rpc error: code = NotFound desc = could not find container \"5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e\": container with ID starting with 5c5c22def24588a89e36b937192ee3c890999e34f590c20bc26323a48188387e not found: ID does not exist"
Oct 31 01:26:18.646853 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.646834    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:26:18.652396 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:18.652377    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-kserve-predictor-67d76765c6-x978w"]
Oct 31 01:26:19.199178 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:19.199151    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" path="/var/lib/kubelet/pods/15cf71be-766c-44e7-891d-183fa9256fa6/volumes"
Oct 31 01:26:19.623746 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:19.623711    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerStarted","Data":"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"}
Oct 31 01:26:19.623931 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:19.623773    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerStarted","Data":"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"}
Oct 31 01:26:19.623988 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:19.623967    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:19.650949 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:19.650906    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podStartSLOduration=2.406291442 podStartE2EDuration="13.650892641s" totalImagesPullingTime="7.577223711s" totalInitContainerRuntime="3.667377488s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:19.650372521 +0000 UTC m=+3442.559375177" watchObservedRunningTime="2026-10-31 01:26:19.650892641 +0000 UTC m=+3442.559895294"
Oct 31 01:26:20.442051 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:20.442004    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:20.581190 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:20.581152    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:20.627715 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:20.627669    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:20.628970 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:20.628943    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:21.631266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:21.631225    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:26.635852 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:26.635822    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:26:26.636340 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:26.636250    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:27.129480 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:27.129442    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:26:36.636959 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:36.636924    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:46.636525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:46.636443    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:48.237888 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:48.237838    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:52.579087 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:52.579049    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:56.636206 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:26:56.636162    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:26:57.128714 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:26:57.128659    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:27:06.636387 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:06.636350    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:27:10.803693 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:10.803634    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:14.165586 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:14.165549    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:14.312579 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:14.312541    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:16.636813 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:16.636771    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:27:25.144835 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:25.144797    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:26.636216 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:26.636177    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:27:27.130241 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:27:27.130205    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:27:36.636239 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:36.636193    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Oct 31 01:27:46.636909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:46.636880    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:27:48.242134 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:48.242105    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:57.129408 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:27:57.129366    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:27:58.350665 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.350633    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:27:58.351176 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.350912    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container" containerID="cri-o://dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5" gracePeriod=30
Oct 31 01:27:58.351176 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.350955    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kube-rbac-proxy" containerID="cri-o://e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a" gracePeriod=30
Oct 31 01:27:58.434047 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.434020    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:58.990862 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.990830    4553 generic.go:356] "Generic (PLEG): container finished" podID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerID="e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a" exitCode=0
Oct 31 01:27:58.991038 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:27:58.990908    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerDied","Data":"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"}
Oct 31 01:28:01.631719 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:01.631655    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Oct 31 01:28:02.485608 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.485585    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:28:02.578266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578199    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\" (UniqueName: \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\") pod \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") "
Oct 31 01:28:02.578266 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578248    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\") pod \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") "
Oct 31 01:28:02.578444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578275    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\") pod \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") "
Oct 31 01:28:02.578444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578306    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\") pod \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") "
Oct 31 01:28:02.578444 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578333    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\") pod \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\" (UID: \"87a7ef60-b6c6-42c4-a723-23eee6c32a41\") "
Oct 31 01:28:02.578652 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578629    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert" pod "87a7ef60-b6c6-42c4-a723-23eee6c32a41" (UID: "87a7ef60-b6c6-42c4-a723-23eee6c32a41"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:28:02.578652 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578636    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location" pod "87a7ef60-b6c6-42c4-a723-23eee6c32a41" (UID: "87a7ef60-b6c6-42c4-a723-23eee6c32a41"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:02.578771 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.578729    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config" pod "87a7ef60-b6c6-42c4-a723-23eee6c32a41" (UID: "87a7ef60-b6c6-42c4-a723-23eee6c32a41"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:28:02.580532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.580505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls" pod "87a7ef60-b6c6-42c4-a723-23eee6c32a41" (UID: "87a7ef60-b6c6-42c4-a723-23eee6c32a41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:28:02.580634 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.580558    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp" pod "87a7ef60-b6c6-42c4-a723-23eee6c32a41" (UID: "87a7ef60-b6c6-42c4-a723-23eee6c32a41"). InnerVolumeSpecName "kube-api-access-tx6gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:28:02.678749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.678716    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6gp\" (UniqueName: \"kubernetes.io/projected/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-api-access-tx6gp\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:02.678749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.678746    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:02.678749 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.678755    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87a7ef60-b6c6-42c4-a723-23eee6c32a41-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:02.679102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.678763    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:02.679102 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:02.678773    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87a7ef60-b6c6-42c4-a723-23eee6c32a41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:03.007666 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.007635    4553 generic.go:356] "Generic (PLEG): container finished" podID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerID="dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5" exitCode=0
Oct 31 01:28:03.007870 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.007695    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerDied","Data":"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"}
Oct 31 01:28:03.007870 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.007707    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"
Oct 31 01:28:03.007870 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.007743    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc" event={"ID":"87a7ef60-b6c6-42c4-a723-23eee6c32a41","Type":"ContainerDied","Data":"977ceafb7264a8c02875f9e159b311ac06250359d71bd511a85808dc44a5285b"}
Oct 31 01:28:03.007870 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.007769    4553 scope.go:122] "RemoveContainer" containerID="e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"
Oct 31 01:28:03.016267 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.016250    4553 scope.go:122] "RemoveContainer" containerID="dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"
Oct 31 01:28:03.023716 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.023698    4553 scope.go:122] "RemoveContainer" containerID="96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"
Oct 31 01:28:03.031179 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.031164    4553 scope.go:122] "RemoveContainer" containerID="e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"
Oct 31 01:28:03.031455 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:28:03.031431    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a\": container with ID starting with e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a not found: ID does not exist" containerID="e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"
Oct 31 01:28:03.031545 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.031461    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a"} err="failed to get container status \"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a\": rpc error: code = NotFound desc = could not find container \"e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a\": container with ID starting with e58bb1c82d194ebbd1875f3528908301edc1fb866ed58266fb5547909f3e135a not found: ID does not exist"
Oct 31 01:28:03.031545 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.031476    4553 scope.go:122] "RemoveContainer" containerID="dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"
Oct 31 01:28:03.031815 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:28:03.031760    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5\": container with ID starting with dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5 not found: ID does not exist" containerID="dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"
Oct 31 01:28:03.031815 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.031804    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5"} err="failed to get container status \"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5\": rpc error: code = NotFound desc = could not find container \"dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5\": container with ID starting with dbb7f79cdb87bc196a6761436b14eb39bdb4d1eb500f4c793c8559a5827807f5 not found: ID does not exist"
Oct 31 01:28:03.031963 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.031827    4553 scope.go:122] "RemoveContainer" containerID="96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"
Oct 31 01:28:03.032118 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:28:03.032088    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08\": container with ID starting with 96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08 not found: ID does not exist" containerID="96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"
Oct 31 01:28:03.032244 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.032124    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08"} err="failed to get container status \"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08\": rpc error: code = NotFound desc = could not find container \"96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08\": container with ID starting with 96e36edf53faa37078aa7f36b39b2af1f5a09b9364b92a4ec0da35c6cd329a08 not found: ID does not exist"
Oct 31 01:28:03.033427 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.033409    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:28:03.039993 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.039974    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-predictor-86668677f7-4zldc"]
Oct 31 01:28:03.203601 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.200375    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" path="/var/lib/kubelet/pods/87a7ef60-b6c6-42c4-a723-23eee6c32a41/volumes"
Oct 31 01:28:03.837134 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837099    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837387    4553 cpu_manager.go:414] "containerMap: removing container" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kube-rbac-proxy"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837398    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kube-rbac-proxy"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837410    4553 cpu_manager.go:414] "containerMap: removing container" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837415    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837421    4553 cpu_manager.go:414] "containerMap: removing container" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837427    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837447    4553 cpu_manager.go:414] "containerMap: removing container" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="storage-initializer"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837451    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="storage-initializer"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837460    4553 cpu_manager.go:414] "containerMap: removing container" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="storage-initializer"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837465    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="storage-initializer"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837471    4553 cpu_manager.go:414] "containerMap: removing container" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kube-rbac-proxy"
Oct 31 01:28:03.837518 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837476    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kube-rbac-proxy"
Oct 31 01:28:03.837909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837543    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kube-rbac-proxy"
Oct 31 01:28:03.837909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837551    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cf71be-766c-44e7-891d-183fa9256fa6" containerName="kserve-container"
Oct 31 01:28:03.837909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837557    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kserve-container"
Oct 31 01:28:03.837909 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.837566    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a7ef60-b6c6-42c4-a723-23eee6c32a41" containerName="kube-rbac-proxy"
Oct 31 01:28:03.840451 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.840410    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:03.841876 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.841860    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.848738 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848716    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:28:03.848845 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848780    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\""
Oct 31 01:28:03.848922 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848844    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Oct 31 01:28:03.848922 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848853    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\""
Oct 31 01:28:03.848922 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848723    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\""
Oct 31 01:28:03.848922 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848786    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Oct 31 01:28:03.849068 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.848940    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\""
Oct 31 01:28:03.852760 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.852741    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:28:03.887264 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.887240    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.887385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.887271    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.887385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.887290    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.887385 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.887314    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.887492 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.887418    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqlm\" (UniqueName: \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988317 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988290    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988317 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988319    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988337    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988356    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988373    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqlm\" (UniqueName: \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.988812 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.988785    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.989082 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.989062    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.989151 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.989085    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:03.991034 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:03.991013    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:04.003356 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:04.003335    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqlm\" (UniqueName: \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\") pod \"isvc-pmml-runtime-predictor-57455d8f9f-lqxqv\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") " pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:04.030526 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:04.030496    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:04.158931 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:04.158890    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:04.434230 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:04.434205    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:28:04.436866 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:28:04.436826    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632539c8_ad8c_4374_9348_fcb77dabcb7b.slice/crio-1ce5c2aaa47421258035fadab1811db6234a2c43a5b441e40011784605b39986 WatchSource:0}: Error finding container 1ce5c2aaa47421258035fadab1811db6234a2c43a5b441e40011784605b39986: Status 404 returned error can't find the container with id 1ce5c2aaa47421258035fadab1811db6234a2c43a5b441e40011784605b39986
Oct 31 01:28:05.018161 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:05.018128    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerStarted","Data":"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"}
Oct 31 01:28:05.018525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:05.018184    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerStarted","Data":"1ce5c2aaa47421258035fadab1811db6234a2c43a5b441e40011784605b39986"}
Oct 31 01:28:09.033752 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:09.033719    4553 generic.go:356] "Generic (PLEG): container finished" podID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerID="e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1" exitCode=0
Oct 31 01:28:09.034175 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:09.033792    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerDied","Data":"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"}
Oct 31 01:28:10.044033 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.043995    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerStarted","Data":"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"}
Oct 31 01:28:10.044427 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.044082    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerStarted","Data":"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"}
Oct 31 01:28:10.044427 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.044413    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:10.044505 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.044465    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:10.045788 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.045763    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:10.072865 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:10.072819    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podStartSLOduration=3.395066216 podStartE2EDuration="7.072808494s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677742278s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:10.071134275 +0000 UTC m=+3552.980136930" watchObservedRunningTime="2026-10-31 01:28:10.072808494 +0000 UTC m=+3552.981811147"
Oct 31 01:28:11.048376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:11.048336    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:16.054599 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:16.054568    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:28:16.055095 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:16.055067    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:26.055723 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:26.055670    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:27.130876 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:28:27.130838    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:28:34.479401 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:34.479363    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:36.055376 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:36.055334    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:46.055724 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:46.055664    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:48.337465 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:48.337416    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:56.055523 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:56.055482    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:28:57.128908 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:28:57.128865    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:28:57.474216 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:57.474184    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:28:57.479239 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:28:57.479218    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmgdp_0e79d059-a2d5-419e-bb3a-de74bb0082e5/ovn-acl-logging/0.log"
Oct 31 01:29:06.055027 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:06.054990    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:29:07.188588 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:07.188546    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:16.055517 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:16.055476    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:29:19.780542 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:19.780502    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:19.927635 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:19.927581    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:20.195723 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:20.195688    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:29:27.130048 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:29:27.130012    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:29:30.196178 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:30.196137    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Oct 31 01:29:40.196906 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:40.196816    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:29:45.961067 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:45.961035    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:29:45.961478 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:45.961259    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container" containerID="cri-o://6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6" gracePeriod=30
Oct 31 01:29:45.961478 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:45.961350    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kube-rbac-proxy" containerID="cri-o://b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb" gracePeriod=30
Oct 31 01:29:46.043234 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:46.043192    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:46.049166 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:46.049141    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Oct 31 01:29:46.392896 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:46.392817    4553 generic.go:356] "Generic (PLEG): container finished" podID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerID="b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb" exitCode=0
Oct 31 01:29:46.393038 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:46.392891    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerDied","Data":"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"}
Oct 31 01:29:48.408091 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:48.408063    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:49.906925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.906902    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:29:49.986839 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.986805    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\") pod \"632539c8-ad8c-4374-9348-fcb77dabcb7b\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") "
Oct 31 01:29:49.986969 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.986844    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\") pod \"632539c8-ad8c-4374-9348-fcb77dabcb7b\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") "
Oct 31 01:29:49.986969 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.986876    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\") pod \"632539c8-ad8c-4374-9348-fcb77dabcb7b\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") "
Oct 31 01:29:49.986969 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.986898    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\" (UniqueName: \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\") pod \"632539c8-ad8c-4374-9348-fcb77dabcb7b\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") "
Oct 31 01:29:49.986969 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.986929    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\") pod \"632539c8-ad8c-4374-9348-fcb77dabcb7b\" (UID: \"632539c8-ad8c-4374-9348-fcb77dabcb7b\") "
Oct 31 01:29:49.987288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.987258    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config" pod "632539c8-ad8c-4374-9348-fcb77dabcb7b" (UID: "632539c8-ad8c-4374-9348-fcb77dabcb7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:49.987288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.987268    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert" pod "632539c8-ad8c-4374-9348-fcb77dabcb7b" (UID: "632539c8-ad8c-4374-9348-fcb77dabcb7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:49.987288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.987280    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location" pod "632539c8-ad8c-4374-9348-fcb77dabcb7b" (UID: "632539c8-ad8c-4374-9348-fcb77dabcb7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:49.989106 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.989080    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls" pod "632539c8-ad8c-4374-9348-fcb77dabcb7b" (UID: "632539c8-ad8c-4374-9348-fcb77dabcb7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:29:49.989186 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:49.989168    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm" pod "632539c8-ad8c-4374-9348-fcb77dabcb7b" (UID: "632539c8-ad8c-4374-9348-fcb77dabcb7b"). InnerVolumeSpecName "kube-api-access-2fqlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:50.088149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.088076    4553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632539c8-ad8c-4374-9348-fcb77dabcb7b-kserve-provision-location\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:50.088149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.088106    4553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-cabundle-cert\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:50.088149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.088121    4553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:50.088149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.088135    4553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632539c8-ad8c-4374-9348-fcb77dabcb7b-proxy-tls\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:50.088149 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.088149    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqlm\" (UniqueName: \"kubernetes.io/projected/632539c8-ad8c-4374-9348-fcb77dabcb7b-kube-api-access-2fqlm\") on node \"ip-10-0-135-193.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:50.408058 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.408030    4553 generic.go:356] "Generic (PLEG): container finished" podID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerID="6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6" exitCode=0
Oct 31 01:29:50.408233 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.408099    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"
Oct 31 01:29:50.408233 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.408109    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerDied","Data":"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"}
Oct 31 01:29:50.408233 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.408178    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv" event={"ID":"632539c8-ad8c-4374-9348-fcb77dabcb7b","Type":"ContainerDied","Data":"1ce5c2aaa47421258035fadab1811db6234a2c43a5b441e40011784605b39986"}
Oct 31 01:29:50.408233 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.408217    4553 scope.go:122] "RemoveContainer" containerID="b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"
Oct 31 01:29:50.417549 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.417528    4553 scope.go:122] "RemoveContainer" containerID="6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"
Oct 31 01:29:50.425075 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.425059    4553 scope.go:122] "RemoveContainer" containerID="e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"
Oct 31 01:29:50.431885 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.431864    4553 scope.go:122] "RemoveContainer" containerID="b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"
Oct 31 01:29:50.432168 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:29:50.432140    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb\": container with ID starting with b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb not found: ID does not exist" containerID="b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"
Oct 31 01:29:50.432268 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.432178    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb"} err="failed to get container status \"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb\": rpc error: code = NotFound desc = could not find container \"b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb\": container with ID starting with b98ac01644042b1032c9503cc8e2da2de580e3a5e3eb0b799d6f9a9cb02ed2cb not found: ID does not exist"
Oct 31 01:29:50.432268 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.432199    4553 scope.go:122] "RemoveContainer" containerID="6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"
Oct 31 01:29:50.432480 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:29:50.432460    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6\": container with ID starting with 6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6 not found: ID does not exist" containerID="6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"
Oct 31 01:29:50.432527 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.432487    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6"} err="failed to get container status \"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6\": rpc error: code = NotFound desc = could not find container \"6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6\": container with ID starting with 6dde48ee6f86636849c635e3a753752beae62f1619716964cec62f42a56dcfb6 not found: ID does not exist"
Oct 31 01:29:50.432527 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.432503    4553 scope.go:122] "RemoveContainer" containerID="e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"
Oct 31 01:29:50.432765 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:29:50.432739    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1\": container with ID starting with e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1 not found: ID does not exist" containerID="e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"
Oct 31 01:29:50.432837 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.432771    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1"} err="failed to get container status \"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1\": rpc error: code = NotFound desc = could not find container \"e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1\": container with ID starting with e560ecdd1ffbf4d6c359edd0c49cccedfed9dcf25ee13cb31bea84732dd226f1 not found: ID does not exist"
Oct 31 01:29:50.434913 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.434894    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:29:50.442472 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:50.442453    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-runtime-predictor-57455d8f9f-lqxqv"]
Oct 31 01:29:51.195299 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:51.195257    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:51.199480 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:51.199455    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" path="/var/lib/kubelet/pods/632539c8-ad8c-4374-9348-fcb77dabcb7b/volumes"
Oct 31 01:29:53.061570 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:53.061526    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:53.208897 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:29:53.208865    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:57.129244 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:29:57.129208    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:30:03.024716 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:03.024659    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:03.163653 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:03.163609    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:21.728288 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.728247    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:21.989398 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989323    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-whtv4/must-gather-q27qw"]
Oct 31 01:30:21.989719 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989700    4553 cpu_manager.go:414] "containerMap: removing container" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="storage-initializer"
Oct 31 01:30:21.989767 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989723    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="storage-initializer"
Oct 31 01:30:21.989803 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989767    4553 cpu_manager.go:414] "containerMap: removing container" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kube-rbac-proxy"
Oct 31 01:30:21.989803 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989775    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kube-rbac-proxy"
Oct 31 01:30:21.989803 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989786    4553 cpu_manager.go:414] "containerMap: removing container" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container"
Oct 31 01:30:21.989803 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989795    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container"
Oct 31 01:30:21.989925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989881    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kube-rbac-proxy"
Oct 31 01:30:21.989925 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.989890    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="632539c8-ad8c-4374-9348-fcb77dabcb7b" containerName="kserve-container"
Oct 31 01:30:21.993080 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.993061    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:21.999611 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.999593    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whtv4\"/\"openshift-service-ca.crt\""
Oct 31 01:30:21.999739 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.999613    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whtv4\"/\"kube-root-ca.crt\""
Oct 31 01:30:21.999739 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:21.999658    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-whtv4\"/\"default-dockercfg-94wtk\""
Oct 31 01:30:22.001981 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.001947    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/must-gather-q27qw"]
Oct 31 01:30:22.019660 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.019603    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:22.119422 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.119397    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b34df8d6-1229-4418-be97-9bfda4c1bed4-must-gather-output\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.119540 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.119457    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdl2c\" (UniqueName: \"kubernetes.io/projected/b34df8d6-1229-4418-be97-9bfda4c1bed4-kube-api-access-qdl2c\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.150514 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.150488    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:22.220787 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.220760    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdl2c\" (UniqueName: \"kubernetes.io/projected/b34df8d6-1229-4418-be97-9bfda4c1bed4-kube-api-access-qdl2c\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.220894 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.220813    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b34df8d6-1229-4418-be97-9bfda4c1bed4-must-gather-output\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.221133 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.221111    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b34df8d6-1229-4418-be97-9bfda4c1bed4-must-gather-output\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.235603 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.235580    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdl2c\" (UniqueName: \"kubernetes.io/projected/b34df8d6-1229-4418-be97-9bfda4c1bed4-kube-api-access-qdl2c\") pod \"must-gather-q27qw\" (UID: \"b34df8d6-1229-4418-be97-9bfda4c1bed4\") " pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.326458 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.326410    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/must-gather-q27qw"
Oct 31 01:30:22.598571 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:22.598481    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/must-gather-q27qw"]
Oct 31 01:30:22.601471 ip-10-0-135-193 kubenswrapper[4553]: W1031 01:30:22.601445    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb34df8d6_1229_4418_be97_9bfda4c1bed4.slice/crio-2c203bacc6c94c63a23f248a844b9b0935eedf24f8fcadddafe50706514c9c11 WatchSource:0}: Error finding container 2c203bacc6c94c63a23f248a844b9b0935eedf24f8fcadddafe50706514c9c11: Status 404 returned error can't find the container with id 2c203bacc6c94c63a23f248a844b9b0935eedf24f8fcadddafe50706514c9c11
Oct 31 01:30:23.525257 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:23.525231    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/must-gather-q27qw" event={"ID":"b34df8d6-1229-4418-be97-9bfda4c1bed4","Type":"ContainerStarted","Data":"2c203bacc6c94c63a23f248a844b9b0935eedf24f8fcadddafe50706514c9c11"}
Oct 31 01:30:24.531961 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:24.531928    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/must-gather-q27qw" event={"ID":"b34df8d6-1229-4418-be97-9bfda4c1bed4","Type":"ContainerStarted","Data":"37a2e4a0c2306edc61e269020bcd03762869ce6b02d70b8bf8eb62b7a661c6ed"}
Oct 31 01:30:24.532370 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:24.531973    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/must-gather-q27qw" event={"ID":"b34df8d6-1229-4418-be97-9bfda4c1bed4","Type":"ContainerStarted","Data":"a9126e557aaa45fc9ec16507b79e2fb300ccab7df0132854f2d585c95339fd3e"}
Oct 31 01:30:24.556747 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:24.556670    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-whtv4/must-gather-q27qw" podStartSLOduration=2.7229058630000003 podStartE2EDuration="3.55665584s" totalImagesPullingTime="833.749977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:24.55441405 +0000 UTC m=+3687.463416704" watchObservedRunningTime="2026-10-31 01:30:24.55665584 +0000 UTC m=+3687.465658494"
Oct 31 01:30:25.386883 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:25.386837    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2pb9q_cbfee234-053b-4291-aecd-5c197dded041/global-pull-secret-syncer/0.log"
Oct 31 01:30:25.711083 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:25.711053    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log"
Oct 31 01:30:25.913377 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:25.913346    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-193.us-east-2.compute.internal_2b7e31f843ae335a5983ffff48ea08a5/haproxy/0.log"
Oct 31 01:30:27.237721 ip-10-0-135-193 kubenswrapper[4553]: E1031 01:30:27.234327    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnddf" containerName="extractor"
Oct 31 01:30:28.080697 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:28.079835    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:28.236698 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:28.231374    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:31.797689 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:31.797637    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h5kc6_7e1d95e0-dcab-42de-9384-3f1ff3b506b0/node-exporter/0.log"
Oct 31 01:30:31.832341 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:31.832307    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h5kc6_7e1d95e0-dcab-42de-9384-3f1ff3b506b0/kube-rbac-proxy/0.log"
Oct 31 01:30:31.878211 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:31.878182    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h5kc6_7e1d95e0-dcab-42de-9384-3f1ff3b506b0/init-textfile/0.log"
Oct 31 01:30:32.442279 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:32.442246    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m9wb8_854ab973-d307-4302-a733-ac0aae740462/prometheus-operator/0.log"
Oct 31 01:30:32.475203 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:32.475177    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m9wb8_854ab973-d307-4302-a733-ac0aae740462/kube-rbac-proxy/0.log"
Oct 31 01:30:32.514071 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:32.514041    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-b25mx_7b91d7c0-7891-4c5c-af83-100228221cb1/prometheus-operator-admission-webhook/0.log"
Oct 31 01:30:33.154362 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.154331    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"]
Oct 31 01:30:33.158316 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.158057    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.175424 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.175391    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"]
Oct 31 01:30:33.278364 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.278319    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-podres\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.278532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.278382    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-lib-modules\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.278532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.278453    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-sys\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.278532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.278489    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-proc\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.278532 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.278515    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjzf\" (UniqueName: \"kubernetes.io/projected/d0e168bb-ad88-4806-ba36-3897ea506914-kube-api-access-lkjzf\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379289 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379256    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-lib-modules\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379289 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379292    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-sys\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379316    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-proc\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379333    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjzf\" (UniqueName: \"kubernetes.io/projected/d0e168bb-ad88-4806-ba36-3897ea506914-kube-api-access-lkjzf\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379393    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-podres\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379413    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-sys\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379413    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-lib-modules\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379466    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-podres\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.379525 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.379413    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e168bb-ad88-4806-ba36-3897ea506914-proc\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.397307 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.397278    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjzf\" (UniqueName: \"kubernetes.io/projected/d0e168bb-ad88-4806-ba36-3897ea506914-kube-api-access-lkjzf\") pod \"perf-node-gather-daemonset-6zxmh\" (UID: \"d0e168bb-ad88-4806-ba36-3897ea506914\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.475623 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.475551    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:33.758287 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:33.758193    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"]
Oct 31 01:30:34.604759 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:34.603867    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh" event={"ID":"d0e168bb-ad88-4806-ba36-3897ea506914","Type":"ContainerStarted","Data":"5e81c0e75900d7414183766ea4bfe2eb7aad228cb46e205c4b132614d66ac2ea"}
Oct 31 01:30:34.604759 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:34.603943    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh" event={"ID":"d0e168bb-ad88-4806-ba36-3897ea506914","Type":"ContainerStarted","Data":"d2eefc0af2757c02da66cdc25c4b87fc9672e3aee776d9b8207aa95cd9c539d1"}
Oct 31 01:30:34.604759 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:34.603980    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:34.628460 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:34.628406    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh" podStartSLOduration=1.628392803 podStartE2EDuration="1.628392803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:34.6280623 +0000 UTC m=+3697.537064954" watchObservedRunningTime="2026-10-31 01:30:34.628392803 +0000 UTC m=+3697.537395457"
Oct 31 01:30:35.031544 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:35.031509    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d4d58bbdf-tvzxk_247590ce-7250-44d4-9534-634352b1cd92/console/0.log"
Oct 31 01:30:35.087219 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:35.087194    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-l7s2h_91ee8dff-fe5b-4c23-a771-ea7282e170f1/download-server/0.log"
Oct 31 01:30:36.879605 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:36.879574    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76d7d_5d383848-ee88-4158-b9f8-8dd88fdab4c4/dns/0.log"
Oct 31 01:30:36.913841 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:36.913816    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76d7d_5d383848-ee88-4158-b9f8-8dd88fdab4c4/kube-rbac-proxy/0.log"
Oct 31 01:30:37.022377 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:37.022350    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ck7z7_22e47f56-d809-4b6a-861b-98b7bd8241fb/dns-node-resolver/0.log"
Oct 31 01:30:37.667069 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:37.667037    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b97446cdb-xckwp_7a5d5943-6aba-4318-8000-62ed7ecf1a39/registry/0.log"
Oct 31 01:30:37.774654 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:37.774619    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjgcf_e5ba2243-f40b-4f95-a955-bd82c885d457/node-ca/0.log"
Oct 31 01:30:39.202489 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:39.202449    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jrkck_c3e796e1-0c4b-455c-906d-98ee4c141fe0/serve-healthcheck-canary/0.log"
Oct 31 01:30:39.831849 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:39.831822    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnddf_bf22e752-9631-48b6-94da-3e7c458d7601/kube-rbac-proxy/0.log"
Oct 31 01:30:39.868431 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:39.868410    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnddf_bf22e752-9631-48b6-94da-3e7c458d7601/exporter/0.log"
Oct 31 01:30:39.905926 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:39.905899    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnddf_bf22e752-9631-48b6-94da-3e7c458d7601/extractor/0.log"
Oct 31 01:30:40.069355 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:40.069322    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-2qbts-74vm8_4cb2c039-eb2d-4b2e-bdc0-5c446e609a92/insights-gathering/0.log"
Oct 31 01:30:40.619375 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:40.619349    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-6zxmh"
Oct 31 01:30:43.230175 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:43.230146    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-56ddc4c448-wm8d8_1ea0029f-543e-4002-a06e-30330910b046/server/0.log"
Oct 31 01:30:44.011137 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:44.011099    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7f9f8cdc65-z4k4j_c12012ed-939e-4297-a548-8d688891babd/manager/0.log"
Oct 31 01:30:44.313451 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:44.313371    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-78bb9c6ff8-ggkf9_31742dd9-374b-4a4f-846b-7e148923c08f/seaweedfs/0.log"
Oct 31 01:30:48.458202 ip-10-0-135-193 kubenswrapper[4553]: I1031 01:30:48.458145    4553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-42vpj_25f0fe6f-7a3d-4966-95be-31c2ee3941a6/konnectivity-agent/0.log" limit=1024