Oct 31 07:28:36.667485 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:28:37.071766 ip-10-0-137-69 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:37.071766 ip-10-0-137-69 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:28:37.071766 ip-10-0-137-69 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:37.071766 ip-10-0-137-69 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:37.077999 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.077905    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.077999 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.077999    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.077999 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078002    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.077999 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078006    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078008    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078011    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078014    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078017    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078020    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078023    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078025    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078028    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078031    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078034    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078037    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078039    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078042    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078044    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078047    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078050    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078052    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078056    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078058    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.078138 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078061    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078064    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078067    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078071    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078076    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078079    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078081    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078084    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078086    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078089    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078091    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078094    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078097    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078100    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078102    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078105    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078108    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078111    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078114    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078129    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.078649 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078132    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078135    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078138    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078141    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078143    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078146    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078149    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078152    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078155    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078158    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078161    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078163    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078166    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078169    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078171    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078174    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078177    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078179    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078182    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078185    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.079131 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078187    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078190    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078192    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078195    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078198    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078201    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078204    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078209    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078213    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078216    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078219    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078222    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078227    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078230    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078233    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078236    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078238    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.079657 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078241    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078243    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078246    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078248    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078251    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078253    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078256    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078258    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078261    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078263    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078266    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078268    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078271    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078274    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078277    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078279    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078282    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078284    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078287    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.080094 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078290    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078293    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078296    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078299    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078303    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.078306    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079138    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079146    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079149    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079152    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079155    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079160    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079165    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079168    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079172    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079174    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079177    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079180    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079182    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.080574 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079185    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079188    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079190    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079193    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079196    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079198    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079201    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079204    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079206    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079209    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079211    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079214    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079218    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079220    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079224    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079226    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079229    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079232    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079235    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079237    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.081028 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079240    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079243    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079245    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079248    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079250    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079254    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079256    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079259    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079261    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079265    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079270    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079274    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079277    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079280    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079283    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079286    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079289    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079291    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079293    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079296    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.081520 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079298    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079301    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079304    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079306    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079310    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079312    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079315    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079318    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079320    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079323    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079326    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079329    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079331    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079334    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079336    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079339    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079342    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079344    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079347    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079350    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.082005 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079353    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079355    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079358    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079361    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079363    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079366    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079368    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079371    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079374    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079377    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079380    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079382    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079385    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079387    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079390    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079393    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079396    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079398    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079401    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.082535 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079403    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079406    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079408    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079411    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079413    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079416    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079418    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079421    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079424    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079428    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079431    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079433    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079436    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079439    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079949    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079955    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079958    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079961    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079964    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079967    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.082988 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079970    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079972    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079975    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079978    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079980    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079983    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079986    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079989    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079991    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079994    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079997    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.079999    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080004    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080008    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080010    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080013    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080016    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080018    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080021    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080023    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.083477 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080026    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080028    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080031    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080033    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080036    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080039    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080042    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080045    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080047    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080050    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080053    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080056    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080058    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080061    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080063    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080066    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080068    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080071    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080073    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080076    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.083959 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080078    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080081    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080083    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080086    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080089    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080093    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080096    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080099    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080102    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080105    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080107    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080110    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080112    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080115    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080132    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080136    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080140    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080145    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080148    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.084534 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080152    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080154    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080157    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080160    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080162    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080165    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080168    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080171    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080174    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080177    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080179    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080182    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080184    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080187    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080190    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080192    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080195    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080197    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080200    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080202    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.084980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080205    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080207    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080210    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080212    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080215    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080218    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080220    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080222    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080227    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080230    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080233    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080240    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080243    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080245    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080248    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080251    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080254    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080257    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.085475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080260    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.080262    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081379    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081393    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081402    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081407    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081412    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081415    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081419    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081424    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081427    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081430    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081434    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081437    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081441    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081444    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081447    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081450    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081453    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081456    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081461    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081464    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081467    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081470    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:28:37.085916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081474    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081480    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081483    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081486    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081490    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081493    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081496    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081499    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081503    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081507    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081511    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081514    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081517    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081520    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081524    2582 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081527    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081532    2582 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081536    2582 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081540    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081543    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081546    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081551    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081554    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081557    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081560    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:28:37.086496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081563    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081566    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081569    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081572    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081575    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081578    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081581    2582 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081585    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081589    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081593    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081596    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081599    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081602    2582 flags.go:64] FLAG: --help="false"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081605    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081609    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081612    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081614    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081618    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081621    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081625    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081628    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081631    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081634    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:28:37.087109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081637    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081640    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081643    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081646    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081649    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081652    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081655    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081658    2582 flags.go:64] FLAG: --lock-file=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081661    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081664    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081667    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081673    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081676    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081679    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081682    2582 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081686    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081689    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081692    2582 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081695    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081700    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081704    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081707    2582 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081711    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081714    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081717    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:28:37.087688 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081720    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081722    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081725    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081728    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081737    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081740    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081743    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081746    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081748    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081751    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081755    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081757    2582 flags.go:64] FLAG: --port="10250"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081761    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081764    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c5d28f9418ef944c"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081767    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081770    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081773    2582 flags.go:64] FLAG: --register-node="true"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081776    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081781    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081783    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081786    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081789    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081793    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081797    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081800    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081803    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:28:37.088775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081806    2582 flags.go:64] FLAG: --runonce="false"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081810    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081813    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081816    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081819    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081822    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081825    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081828    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081831    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081834    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081837    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081840    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081842    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081846    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081849    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081851    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081857    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081860    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081863    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081868    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081871    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081874    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081876    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081879    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081883    2582 flags.go:64] FLAG: --v="2"
Oct 31 07:28:37.089856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081887    2582 flags.go:64] FLAG: --version="false"
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081892    2582 flags.go:64] FLAG: --vmodule=""
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081895    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.081901    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082038    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082043    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082047    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082049    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082052    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082055    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082058    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082061    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082063    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082066    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082069    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082072    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082074    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082077    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082080    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082082    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.090829 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082086    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082088    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082091    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082093    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082096    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082099    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082101    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082104    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082106    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082109    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082112    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082127    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082131    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082135    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082138    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082144    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082147    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082153    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.091343 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082157    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082161    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082165    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082169    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082172    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082175    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082178    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082180    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082183    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082185    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082188    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082191    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082193    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082196    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082199    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082202    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082204    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082207    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082210    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082212    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.091789 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082215    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082218    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082220    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082223    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082226    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082228    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082231    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082233    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082236    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082240    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082243    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082247    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082250    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082253    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082255    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082258    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082260    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082264    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082266    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.092279 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082269    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082272    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082274    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082277    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082279    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082282    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082284    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082287    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082289    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082292    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082294    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082297    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082299    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082302    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082305    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082307    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082311    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082313    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082316    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082319    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.092773 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082321    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082324    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082328    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082330    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082334    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082337    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082340    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082342    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082345    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082347    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082350    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082353    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082355    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.082382    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082420    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082425    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.093254 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082428    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082431    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082434    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082437    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082440    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082442    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082445    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082448    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082450    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082453    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082455    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082458    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082461    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082464    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082466    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082469    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082472    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082474    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082477    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.093639 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082481    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082484    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082487    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082490    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082492    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082495    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082499    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082503    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082506    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082509    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082512    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082515    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082518    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082522    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082526    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082530    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082533    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082536    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082539    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.094095 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082541    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082544    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082547    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082550    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082552    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082555    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082558    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082561    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082564    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082566    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082569    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082572    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082575    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082578    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082582    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082585    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082587    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082590    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082592    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082595    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.094618 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082598    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082600    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082603    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082606    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082608    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082612    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082615    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082617    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082620    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082622    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082625    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082628    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082630    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082633    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082635    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082638    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082640    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082643    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082646    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.095092 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082648    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082651    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082653    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082656    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082658    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082661    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082664    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082668    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082671    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082674    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082677    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082680    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082683    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082685    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082688    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082690    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082693    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082696    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082699    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082701    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.095812 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082703    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082706    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082709    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082711    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082714    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082716    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.082719    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.090982    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.091001    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.091018    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.091023    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091083    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091089    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091093    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091097    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091100    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091102    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091105    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091109    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.096322 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091112    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091115    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091135    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091139    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091143    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091147    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091149    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091153    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091155    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091158    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091161    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091164    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091166    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091169    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091172    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091174    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091177    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091180    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091182    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.096807 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091190    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091193    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091196    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091199    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091201    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091204    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091206    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091209    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091212    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091214    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091217    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091219    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091222    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091225    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091228    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091231    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091234    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091237    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091239    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.097349 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091242    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091245    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091247    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091250    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091252    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091255    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091258    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091260    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091263    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091267    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091272    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091275    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091278    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091282    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091284    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091287    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091290    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091293    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091296    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.098103 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091298    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091301    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091303    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091306    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091309    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091312    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091314    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091317    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091319    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091322    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091325    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091327    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091330    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091332    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091336    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091338    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091341    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091344    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091346    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091349    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.098750 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091352    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091355    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091358    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091361    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091364    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091367    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091369    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091372    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091375    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091377    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091380    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091383    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091385    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091388    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091391    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091393    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091396    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091398    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091401    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091404    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.099351 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091407    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.091436    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091598    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091603    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091606    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091612    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091616    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091619    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091621    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091624    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091627    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091629    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091632    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091635    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.099914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091638    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091641    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091644    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091646    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091649    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091652    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091655    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091658    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091661    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091663    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091666    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091669    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091671    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091674    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091676    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091679    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091682    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091685    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091687    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.100455 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091690    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091692    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091695    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091698    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091700    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091704    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091709    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091712    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091715    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091717    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091720    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091722    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091725    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091728    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091730    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091733    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091736    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091740    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091743    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.100980 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091746    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091749    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091751    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091754    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091756    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091759    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091761    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091763    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091766    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091768    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091771    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091773    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091776    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091778    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091781    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091783    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091786    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091789    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091791    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091794    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.101460 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091798    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091801    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091803    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091806    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091808    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091811    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091814    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091816    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091819    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091821    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091824    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091826    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091829    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091832    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091834    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091837    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091839    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091842    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091845    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091847    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.102001 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091850    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091852    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091855    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091857    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091859    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091862    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091864    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091867    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091869    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091872    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091875    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091877    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091880    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091883    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091885    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.091908    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:37.102505 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091941    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091944    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091947    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091950    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091953    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091956    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091959    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091962    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091965    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091968    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091971    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091973    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091976    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091979    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091985    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091987    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091990    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091993    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091996    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.091998    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:37.102886 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092001    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092003    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092006    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092009    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092011    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092014    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092016    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092019    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092022    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092024    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092027    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092030    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092033    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092035    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092038    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092040    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092043    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092046    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092048    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:37.103425 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092051    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092053    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092056    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092058    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092061    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092064    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092066    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092069    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092072    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092074    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092078    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092082    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092085    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092088    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092090    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092093    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092095    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092098    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092101    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:37.103872 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092103    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092106    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092109    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092113    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092129    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092135    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092149    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092152    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092155    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092159    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092162    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092165    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092167    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092170    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092172    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092175    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092177    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092180    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092182    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092185    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:37.104339 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092187    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092191    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092193    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092196    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092198    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092201    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092203    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092206    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092208    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092211    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092213    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092216    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092218    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092220    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092223    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092225    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092228    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092230    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092235    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092237    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:37.104898 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092240    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092245    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092248    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092250    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092253    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092255    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092258    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:37.092260    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.092326    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.094337    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.095378    2582 server.go:1014] "Starting client certificate rotation"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.095488    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.095545    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.097403    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:37.105567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.100748    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:37.112417 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.112398    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:28:37.115749 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.115734    2582 log.go:25] "Validated CRI v1 image API"
Oct 31 07:28:37.116576 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.116549    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:28:37.123807 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.123787    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:37.124536 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.124514    2582 fs.go:135] Filesystem UUIDs: map[330852c8-a011-423b-96b6-f2566b82091d:/dev/nvme0n1p3 557a13b1-392c-495d-9043-042406b724f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 07:28:37.124613 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.124537    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 07:28:37.131137 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.130996    2582 manager.go:217] Machine: {Timestamp:2026-10-31 07:28:37.129608038 +0000 UTC m=+0.342294244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095446 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c559277247202126e6cdbb829b90f SystemUUID:ec2c5592-7724-7202-126e-6cdbb829b90f BootID:2d54593a-85b7-4f4b-b69e-aa4943c92b4e Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:e3:c9:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:e3:c9:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:32:6b:55:db:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:28:37.131597 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.131586    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:28:37.131696 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.131683    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:28:37.135233 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135014    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:28:37.135378 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135235    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:28:37.135447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135380    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:28:37.135447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135389    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:28:37.135447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135425    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:37.135447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.135444    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:37.136232 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.136221    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:28:37.139947 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.139935    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:28:37.140071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140061    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:28:37.140109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140095    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:28:37.140109 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140105    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:28:37.140191 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140115    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:28:37.140191 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140139    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:28:37.140191 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140149    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:28:37.140482 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.140470    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:37.142335 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.142308    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kft87"
Oct 31 07:28:37.142765 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.142743    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:28:37.147803 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.147786    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:37.148581 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148567    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:28:37.148630 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148599    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:28:37.148806 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148794    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:28:37.148839 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148814    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:28:37.148839 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148821    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:28:37.148839 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148827    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:28:37.148839 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148837    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:28:37.148935 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.148843    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:28:37.149380 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149362    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:28:37.149380 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149375    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:28:37.149380 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149383    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:28:37.149626 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149389    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:28:37.149626 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149398    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:28:37.149626 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149407    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:28:37.149885 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149860    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kft87"
Oct 31 07:28:37.149935 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149887    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:28:37.149935 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.149897    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:28:37.154056 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.154042    2582 server.go:1263] "Started kubelet"
Oct 31 07:28:37.154168 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.154092    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:28:37.154263 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.154208    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:28:37.154330 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.154286    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:28:37.155873 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.155854    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:28:37.155861 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:28:37.157243 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.157229    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:28:37.160760 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.160724    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:37.161136 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.161102    2582 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-137-69.us-east-2.compute.internal" not found
Oct 31 07:28:37.161817 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.161800    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 07:28:37.162010 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.161994    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:37.162134 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.162098    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:37.163081 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.163058    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:28:37.166282 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.166258    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.us-east-2.compute.internal\" not found"
Oct 31 07:28:37.166396 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.166376    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:28:37.166861 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.166843    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:28:37.166861 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.166864    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:28:37.167051 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167031    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:28:37.167112 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167053    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:28:37.167112 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167093    2582 factory.go:55] Registering systemd factory
Oct 31 07:28:37.167112 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167110    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:28:37.167402 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167381    2582 factory.go:153] Registering CRI-O factory
Oct 31 07:28:37.167402 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167398    2582 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:28:37.167513 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167462    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:28:37.167513 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167498    2582 factory.go:103] Registering Raw factory
Oct 31 07:28:37.167616 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.167518    2582 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:28:37.169473 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.169446    2582 manager.go:319] Starting recovery of all containers
Oct 31 07:28:37.169473 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.169456    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:37.170907 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.170885    2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-69.us-east-2.compute.internal\" not found" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.174112 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.174090    2582 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-137-69.us-east-2.compute.internal" not found
Oct 31 07:28:37.179932 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.179913    2582 manager.go:324] Recovery completed
Oct 31 07:28:37.181366 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.181346    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 07:28:37.184279 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.184264    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.186655 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.186638    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.186736 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.186674    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.186736 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.186686    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.187356 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.187337    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:28:37.187356 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.187350    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:28:37.187486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.187360    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.187486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.187375    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:28:37.188101 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.188085    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.188187 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.188129    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.188187 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.188145    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.189720 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.189707    2582 policy_none.go:50] "Start"
Oct 31 07:28:37.189778 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.189724    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:28:37.189778 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.189738    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:28:37.191784 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.191766    2582 policy_none.go:44] "Start"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.230622    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.231721    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.231738    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.231749    2582 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-137-69.us-east-2.compute.internal" not found
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.231761    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.231792    2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.234704    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.234903    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.237929    2582 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.237955    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.237963    2582 server.go:87] "Starting device plugin registration server"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.238233    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.238242    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.238374    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.238435    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.238442    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.239029    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:28:37.244563 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.239082    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-69.us-east-2.compute.internal\" not found"
Oct 31 07:28:37.332214 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.332093    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"]
Oct 31 07:28:37.332364 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.332221    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.333300 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.333278    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.333398 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.333315    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.333398 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.333330    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.335505 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.335488    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.335679 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.335662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.335729 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.335696    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.336335 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336317    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.336432 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336347    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.336432 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336358    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.336432 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336321    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.336432 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336418    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.336432 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.336428    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.338511 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.338496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.338608 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.338519    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.338930 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.338914    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:37.339261 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339247    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.339355 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339276    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.339355 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339291    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.339565 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339552    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:37.339620 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339598    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:37.339620 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339611    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:37.339677 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.339629    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.347420 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.347269    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.363691 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.363674    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.365611 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.365597    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.367001 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.366986    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.368787 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.368767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.368880 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.368798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.368880 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.368825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.375232 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.375218    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:37.381031 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.381015    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:37.387458 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.387445    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:37.387514 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.387479    2582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.387514 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.387491    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.405027 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.405005    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:37.405101 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:37.405048    2582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469080 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469239 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469239 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469239 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469239 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.469239 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.469191    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.679671 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.679574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:37.684516 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:37.684482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:28:40.118169 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.118082    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 07:28:40.118672 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.118247    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:40.118672 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.118257    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:40.118672 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.118288    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:40.162775 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.162725    2582 apiserver.go:52] "Watching apiserver"
Oct 31 07:28:40.171827 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.171803    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:28:40.172116 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.172100    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-xg255","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal","openshift-image-registry/node-ca-kkwsp","kube-system/global-pull-secret-syncer-xgqwg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd","openshift-multus/multus-additional-cni-plugins-gxcql","openshift-multus/multus-gt6sx","openshift-network-diagnostics/network-check-target-56cfh","kube-system/konnectivity-agent-5dvbr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-6glq2","openshift-network-operator/iptables-alerter-89wm7","openshift-ovn-kubernetes/ovnkube-node-lb4zp","openshift-cluster-node-tuning-operator/tuned-clpj5"]
Oct 31 07:28:40.174380 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.174357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.176398 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.176351    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:39 +0000 UTC" deadline="2028-06-27 18:56:35.325511822 +0000 UTC"
Oct 31 07:28:40.176398 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.176396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.176537 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.176397    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14531h27m55.149118369s"
Oct 31 07:28:40.177624 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.177600    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.177733 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.177706    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.177808 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.177796    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9r8qc\""
Oct 31 07:28:40.178340 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.178323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.178406 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.178381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:40.178995 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.178975    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7sl8v\""
Oct 31 07:28:40.179079 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.179017    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.179079 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.179029    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.179261 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.179249    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:28:40.181096 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.181081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.183846 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.183829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.184159 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.184139    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.184910 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.184895    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:40.186005 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.185986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.188236 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.188218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:40.188316 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.188290    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:40.190676 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.190655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.193211 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.193192    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.193308 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.193266    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:40.194032 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194119 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.194119 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.194226 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.194226 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.194226 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.194226 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8g9w\" (UniqueName: \"kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194421 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9g5\" (UniqueName: \"kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194668    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:40.194739 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.194977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.195071 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195612 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195177    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195612 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195612 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195612 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195797 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.195797 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.195899 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.195899 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcgd\" (UniqueName: \"kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.196000 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s4x\" (UniqueName: \"kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.196000 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.195973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvj7\" (UniqueName: \"kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.196094 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.196065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.196141 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.196005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.197799 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.196567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.199547 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.199477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.202066 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.202050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.279378 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279328    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.279661 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279561    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:28:40.279661 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279600    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:28:40.279661 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279603    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.279661 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279608    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.279661 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279639    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9t28b"
Oct 31 07:28:40.279842 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279659    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.279842 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.279688    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:28:40.280743 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.280730    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:28:40.280853 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.280793    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.280934 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.280874    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:28:40.281168 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.281153    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k6p7p\""
Oct 31 07:28:40.281911 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.281897    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6w9r4\""
Oct 31 07:28:40.282029 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.281974    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:28:40.282029 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.281986    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.282029 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.281993    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:28:40.282187 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.282046    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.282187 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.282176    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:28:40.282282 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.282265    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:28:40.282316 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.282288    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:28:40.282316 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.282296    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:28:40.283179 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283154    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:28:40.283252 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283195    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:28:40.283350 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283337    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x8xx7\""
Oct 31 07:28:40.283402 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283380    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ghqlz\""
Oct 31 07:28:40.283402 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283387    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zjbzq\""
Oct 31 07:28:40.283500 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283478    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:28:40.283556 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283492    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nq6dm\""
Oct 31 07:28:40.283556 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.283497    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bn5lf\""
Oct 31 07:28:40.286994 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.286973    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9t28b"
Oct 31 07:28:40.290157 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.290141    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:28:40.296965 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.296943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297074 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.296972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvj7\" (UniqueName: \"kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.297074 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.296987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.297074 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297219 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297094    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.297219 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.297219 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297219 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297219 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297416    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.297447 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297492    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.297894 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297889    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297922    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297958    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297959    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297961    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.297999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298018    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298020    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298037    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298085    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.298095    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.298486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298437    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.298695    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.798668443 +0000 UTC m=+1.988965463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnhv\" (UniqueName: \"kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.298949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcgd\" (UniqueName: \"kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299203    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299213 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s4x\" (UniqueName: \"kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.299786 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8g9w\" (UniqueName: \"kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9g5\" (UniqueName: \"kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstks\" (UniqueName: \"kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.299954    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqtm\" (UniqueName: \"kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpflk\" (UniqueName: \"kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.300278 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300377    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.300805 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.300593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301235 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301235 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.301235 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.301235 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301412 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301412 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301412 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.301412 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.301617 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301419    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.301617 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.301816 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301794    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.302133 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.301915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.302133 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.302133 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302050    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.302318 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302135    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.302318 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302260    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.302408 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302314    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.302408 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302368    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.302551 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302501    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.302614 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.302595    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.303388 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.303370    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.303496 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.303475    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.304828 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.304805    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.306072 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.306051    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvj7\" (UniqueName: \"kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.307924 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.307901    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:40.308021 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.307929    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:40.308021 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.307944    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.308021 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.308001    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.807982255 +0000 UTC m=+1.998279270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.309703 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.309674    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8g9w\" (UniqueName: \"kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.309887 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.309869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9g5\" (UniqueName: \"kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.310015 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.309994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcgd\" (UniqueName: \"kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.310480 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.310459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s4x\" (UniqueName: \"kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.373029 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.372961    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057eabb38a9074e4b218a9c4b8b8759.slice/crio-77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd WatchSource:0}: Error finding container 77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd: Status 404 returned error can't find the container with id 77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd
Oct 31 07:28:40.379222 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.379204    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:28:40.382337 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.382311    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc22959f6c5e0b1bb2a00c01e37014.slice/crio-49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7 WatchSource:0}: Error finding container 49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7: Status 404 returned error can't find the container with id 49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7
Oct 31 07:28:40.401958 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.401926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.401958 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.401960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.401990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402189 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402199    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402249    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402301    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402301    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnhv\" (UniqueName: \"kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402411    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402447    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402542    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.402652 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402619    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402620    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402622    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402662    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402747    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402778    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402817    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403460 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402897    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstks\" (UniqueName: \"kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqtm\" (UniqueName: \"kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpflk\" (UniqueName: \"kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403035    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.402851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403063    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403108    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.403985 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.403342    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403356    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403361    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.403397    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.903378974 +0000 UTC m=+2.093675989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.403474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.404527    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.404640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.404610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.405281 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.405262    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.409505 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.409478    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnhv\" (UniqueName: \"kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.410937 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.410919    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstks\" (UniqueName: \"kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.411078 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.411057    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqtm\" (UniqueName: \"kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.411469 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.411454    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpflk\" (UniqueName: \"kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.502675 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.502647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xg255"
Oct 31 07:28:40.513089 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.513064    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42302c13_04fb_415c_affb_f652ff0be51c.slice/crio-6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95 WatchSource:0}: Error finding container 6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95: Status 404 returned error can't find the container with id 6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95
Oct 31 07:28:40.522791 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.522769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:28:40.531019 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.530994    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89ab622_9b1b_4954_924b_b75da49f162f.slice/crio-2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40 WatchSource:0}: Error finding container 2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40: Status 404 returned error can't find the container with id 2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40
Oct 31 07:28:40.544480 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.544456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:28:40.552586 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.552560    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b12ceb4_d721_4fe8_9df6_644b2023cd74.slice/crio-a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2 WatchSource:0}: Error finding container a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2: Status 404 returned error can't find the container with id a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2
Oct 31 07:28:40.567215 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.567180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:28:40.571095 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.571069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gt6sx"
Oct 31 07:28:40.577360 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.577336    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1503777_02b6_4287_86f7_e1b7094397f2.slice/crio-5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01 WatchSource:0}: Error finding container 5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01: Status 404 returned error can't find the container with id 5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01
Oct 31 07:28:40.579914 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.579894    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd6ee2e_4015_471d_9fe6_b9efc31594ae.slice/crio-5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398 WatchSource:0}: Error finding container 5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398: Status 404 returned error can't find the container with id 5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398
Oct 31 07:28:40.589163 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.589144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:40.598752 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.598706    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67890a7c_54d4_4af5_a01c_fee014b737a8.slice/crio-c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4 WatchSource:0}: Error finding container c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4: Status 404 returned error can't find the container with id c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4
Oct 31 07:28:40.605600 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.605578    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:28:40.614475 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.614431    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49be424d_4007_47e0_97e2_e8b63e9af6eb.slice/crio-ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef WatchSource:0}: Error finding container ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef: Status 404 returned error can't find the container with id ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef
Oct 31 07:28:40.644245 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.644171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:28:40.651366 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.651346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:28:40.653623 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.653546    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073c9d08_d5d6_4a1b_b39a_fc5a2ab21664.slice/crio-f893b6f89b0f6b5dc170b5c69b6bfa7546447e42e0030b81eab7fe696ac3a9fd WatchSource:0}: Error finding container f893b6f89b0f6b5dc170b5c69b6bfa7546447e42e0030b81eab7fe696ac3a9fd: Status 404 returned error can't find the container with id f893b6f89b0f6b5dc170b5c69b6bfa7546447e42e0030b81eab7fe696ac3a9fd
Oct 31 07:28:40.660214 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:28:40.660193    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2484cbbd_3954_4ee8_a0d2_ca0106675dae.slice/crio-b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e WatchSource:0}: Error finding container b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e: Status 404 returned error can't find the container with id b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e
Oct 31 07:28:40.805394 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.805365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:40.805564 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.805469    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.805564 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.805528    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:28:41.805502521 +0000 UTC m=+2.995799520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.906371 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.906290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.906391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906470    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906534    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906555    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906559    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:28:41.906539119 +0000 UTC m=+3.096836120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.906573 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906567    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.907005 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:40.906615    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:41.90660142 +0000 UTC m=+3.096898422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.975741 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:40.975112    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:41.112741 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.112681    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:41.268851 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.268779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:41.269640 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.268868    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:41.269640 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.268921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:41.269640 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.268969    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:41.284936 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.284840    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xg255" event={"ID":"42302c13-04fb-415c-affb-f652ff0be51c","Type":"ContainerStarted","Data":"6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95"}
Oct 31 07:28:41.290908 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.290807    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:40 +0000 UTC" deadline="2028-04-09 01:24:42.338082314 +0000 UTC"
Oct 31 07:28:41.290908 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.290860    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12617h56m1.047226319s"
Oct 31 07:28:41.294956 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.294878    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"7057eabb38a9074e4b218a9c4b8b8759","Type":"ContainerStarted","Data":"77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd"}
Oct 31 07:28:41.311312 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.311176    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e"}
Oct 31 07:28:41.330897 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.330857    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clpj5" event={"ID":"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664","Type":"ContainerStarted","Data":"f893b6f89b0f6b5dc170b5c69b6bfa7546447e42e0030b81eab7fe696ac3a9fd"}
Oct 31 07:28:41.342271 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.340777    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-89wm7" event={"ID":"49be424d-4007-47e0-97e2-e8b63e9af6eb","Type":"ContainerStarted","Data":"ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef"}
Oct 31 07:28:41.362254 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.362127    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerStarted","Data":"5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01"}
Oct 31 07:28:41.364459 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.364383    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkwsp" event={"ID":"b89ab622-9b1b-4954-924b-b75da49f162f","Type":"ContainerStarted","Data":"2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40"}
Oct 31 07:28:41.416776 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.416666    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerStarted","Data":"49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7"}
Oct 31 07:28:41.432581 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.432501    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5dvbr" event={"ID":"67890a7c-54d4-4af5-a01c-fee014b737a8","Type":"ContainerStarted","Data":"c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4"}
Oct 31 07:28:41.444590 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.444537    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gt6sx" event={"ID":"ccd6ee2e-4015-471d-9fe6-b9efc31594ae","Type":"ContainerStarted","Data":"5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398"}
Oct 31 07:28:41.461743 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.461689    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd" event={"ID":"4b12ceb4-d721-4fe8-9df6-644b2023cd74","Type":"ContainerStarted","Data":"a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2"}
Oct 31 07:28:41.515540 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.515380    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:41.817401 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.817294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:41.817635 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.817529    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:41.817635 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.817603    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:28:43.817577895 +0000 UTC m=+5.007874913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.918668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:41.918728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.918869    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.918921    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:28:43.918904585 +0000 UTC m=+5.109201592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.919225    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.919240    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.919249    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:41.919346 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:41.919276    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:43.919267123 +0000 UTC m=+5.109564123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:42.254674 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:42.254595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:42.254829 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:42.254731    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:42.291443 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:42.291377    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:40 +0000 UTC" deadline="2028-07-09 03:44:01.2200711 +0000 UTC"
Oct 31 07:28:42.291443 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:42.291440    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14804h15m18.928635462s"
Oct 31 07:28:43.262681 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:43.262646    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:43.262856 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:43.262690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:43.262856 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.262794    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:43.262967 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.262920    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:43.844448 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:43.844330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:43.845229 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.844502    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:43.845229 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.844623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:28:47.844602949 +0000 UTC m=+9.034899950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:43.945232 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:43.945190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:43.945389 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:43.945248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:43.945468 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.945390    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:43.945468 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.945455    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:28:47.945435892 +0000 UTC m=+9.135732907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:43.945952 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.945926    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:43.945952 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.945955    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:43.946147 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.945975    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:43.946147 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:43.946023    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:47.94600763 +0000 UTC m=+9.136304645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:44.254603 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:44.254324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:44.254772 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:44.254724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:44.485307 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:44.485197    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="6f2fecc072c8e916caab1ad61a9f57769d833654d1920b39375be587c5a8e920" exitCode=0
Oct 31 07:28:44.485307 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:44.485260    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"6f2fecc072c8e916caab1ad61a9f57769d833654d1920b39375be587c5a8e920"}
Oct 31 07:28:45.258231 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.254933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:45.258231 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:45.255070    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:45.258231 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.255463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:45.258231 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:45.255578    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:45.493111 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.493078    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5dvbr" event={"ID":"67890a7c-54d4-4af5-a01c-fee014b737a8","Type":"ContainerStarted","Data":"2d7fe908c1f429765d80159e8c714c9e75e088413dd80f98d3066de73dfb75c6"}
Oct 31 07:28:45.502527 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.502479    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"7057eabb38a9074e4b218a9c4b8b8759","Type":"ContainerStarted","Data":"c378e6ec4b0c4eb27ba818ec45c34d8afb9a26cb6fd9688e5add5aa2ea040ac5"}
Oct 31 07:28:45.507200 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.507144    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5dvbr" podStartSLOduration=2.30178866 podStartE2EDuration="6.507129781s" totalImagesPullingTime="4.205341121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:45.507078687 +0000 UTC m=+6.697375709" watchObservedRunningTime="2026-10-31 07:28:45.507129781 +0000 UTC m=+6.697426804"
Oct 31 07:28:45.514723 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.514643    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ecc22959f6c5e0b1bb2a00c01e37014" containerID="d1c412f6db1732f1caa295a6ea5fc134eb43f4d37f822b8bd1485b7b6d02ba6b" exitCode=0
Oct 31 07:28:45.514723 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.514688    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerDied","Data":"d1c412f6db1732f1caa295a6ea5fc134eb43f4d37f822b8bd1485b7b6d02ba6b"}
Oct 31 07:28:45.520324 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.520255    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal" podStartSLOduration=6.520241743 podStartE2EDuration="6.520241743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:45.519567166 +0000 UTC m=+6.709864189" watchObservedRunningTime="2026-10-31 07:28:45.520241743 +0000 UTC m=+6.710538765"
Oct 31 07:28:45.571230 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.570843    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:45.571845 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:45.571657    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:46.254683 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:46.254652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:46.254927 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:46.254774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:46.524914 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:46.524203    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkwsp" event={"ID":"b89ab622-9b1b-4954-924b-b75da49f162f","Type":"ContainerStarted","Data":"5f0956ab9d14a67c3f2167ed435168452c15db7395450537d54d3933d0c982cd"}
Oct 31 07:28:46.530915 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:46.530407    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerStarted","Data":"26bae6f15b69f418bfd0916b1ae63d8f5b130986025d926f4257e9fba3e5ddf6"}
Oct 31 07:28:46.555072 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:46.555014    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" podStartSLOduration=7.554996402 podStartE2EDuration="7.554996402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.554543883 +0000 UTC m=+7.744840897" watchObservedRunningTime="2026-10-31 07:28:46.554996402 +0000 UTC m=+7.745293426"
Oct 31 07:28:46.555645 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:46.555604    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kkwsp" podStartSLOduration=2.006212388 podStartE2EDuration="7.555593103s" totalImagesPullingTime="5.549380715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.538611761 +0000 UTC m=+7.728908789" watchObservedRunningTime="2026-10-31 07:28:46.555593103 +0000 UTC m=+7.745890128"
Oct 31 07:28:47.256709 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.256616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:47.256861 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.256802    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:47.260578 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.257148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:47.260753 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.257305    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:47.377243 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.377206    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:47.377853 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.377832    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:28:47.542468 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.541569    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd" event={"ID":"4b12ceb4-d721-4fe8-9df6-644b2023cd74","Type":"ContainerStarted","Data":"d532ed4b6528305959ffedaf611aaabb42de5ec9622922679e14c8b402301cde"}
Oct 31 07:28:47.880182 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.879623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:47.880182 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.879739    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:47.880182 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.879791    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.87977806 +0000 UTC m=+17.070075061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.980241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:47.980327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.980470    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.980488    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.980502    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.980571    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.980554331 +0000 UTC m=+17.170851348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.980963    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:47.981042 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:47.981008    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.980993452 +0000 UTC m=+17.171290458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:48.255674 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:48.255094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:48.255674 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:48.255225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:48.498377 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:48.498352    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:28:48.545817 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:48.545786    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd" event={"ID":"4b12ceb4-d721-4fe8-9df6-644b2023cd74","Type":"ContainerStarted","Data":"4166c309e1ba3a0beb0a9d5360f1f527684c7b7ada12a023c447c8dd770b1aeb"}
Oct 31 07:28:49.256791 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.256762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:49.256923 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:49.256848    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:49.256923 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.256900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:49.257005 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:49.256940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:49.266235 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.265693    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:28:48.49837169Z","UUID":"9bed4bf8-a97c-4719-8948-999fe5d89580","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:28:49.271602 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.271575    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:28:49.271753 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.271622    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:28:49.549332 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.549288    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gt6sx" event={"ID":"ccd6ee2e-4015-471d-9fe6-b9efc31594ae","Type":"ContainerStarted","Data":"47d07390fb351dbedd16a5991375bad0c3663dc6a939e94d0a92289fd110bda3"}
Oct 31 07:28:49.553190 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.553150    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xg255" event={"ID":"42302c13-04fb-415c-affb-f652ff0be51c","Type":"ContainerStarted","Data":"fed353b63ce4b26b61accfd1d2a264756c96585dd0e86207ce46e7c35a993d7d"}
Oct 31 07:28:49.555565 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.555533    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="9c6e4eccadde3626134f41570940e8c3fc624a91636ab9ef45f654f8b2a7211b" exitCode=0
Oct 31 07:28:49.555690 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.555587    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"9c6e4eccadde3626134f41570940e8c3fc624a91636ab9ef45f654f8b2a7211b"}
Oct 31 07:28:49.566077 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.566023    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gt6sx" podStartSLOduration=2.414898591 podStartE2EDuration="10.566005334s" totalImagesPullingTime="8.151106743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:49.565258125 +0000 UTC m=+10.755555149" watchObservedRunningTime="2026-10-31 07:28:49.566005334 +0000 UTC m=+10.756302359"
Oct 31 07:28:49.598614 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:49.598559    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xg255" podStartSLOduration=2.174794112 podStartE2EDuration="10.59854102s" totalImagesPullingTime="8.423746908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:49.598190203 +0000 UTC m=+10.788487226" watchObservedRunningTime="2026-10-31 07:28:49.59854102 +0000 UTC m=+10.788838042"
Oct 31 07:28:50.254736 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:50.254671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:50.254933 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:50.254798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:50.560307 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:50.560215    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd" event={"ID":"4b12ceb4-d721-4fe8-9df6-644b2023cd74","Type":"ContainerStarted","Data":"13b537d91ddc8357b14580e315d6db1a21b42fa05e5120a77b3c6ba58e94ec0d"}
Oct 31 07:28:50.564102 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:50.563612    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="94196f6e8c970ad088f7d8aea7136572934eb06d20d3ce198cec3e141bb84ce8" exitCode=0
Oct 31 07:28:50.564102 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:50.563640    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"94196f6e8c970ad088f7d8aea7136572934eb06d20d3ce198cec3e141bb84ce8"}
Oct 31 07:28:50.577802 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:50.577743    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd" podStartSLOduration=2.908126524 podStartE2EDuration="11.57772305s" totalImagesPullingTime="8.669596526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:50.577539902 +0000 UTC m=+11.767836925" watchObservedRunningTime="2026-10-31 07:28:50.57772305 +0000 UTC m=+11.768020074"
Oct 31 07:28:51.255480 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:51.255391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:51.255664 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:51.255393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:51.255664 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:51.255554    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:51.255664 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:51.255610    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:51.568490 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:51.568449    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clpj5" event={"ID":"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664","Type":"ContainerStarted","Data":"c496e4e768b8082e82e607c02d2d7b1b61c2e540c82c6bfe566208eea7ccb868"}
Oct 31 07:28:51.570098 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:51.570067    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-89wm7" event={"ID":"49be424d-4007-47e0-97e2-e8b63e9af6eb","Type":"ContainerStarted","Data":"c88cf6bc38cc2b0139f02765da5f10d3aa95f4ff922d6bef6cac54ed6ee8744a"}
Oct 31 07:28:51.584129 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:51.584077    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-clpj5" podStartSLOduration=2.240990923 podStartE2EDuration="12.584057734s" totalImagesPullingTime="10.343066811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:51.583733885 +0000 UTC m=+12.774030908" watchObservedRunningTime="2026-10-31 07:28:51.584057734 +0000 UTC m=+12.774354756"
Oct 31 07:28:52.254534 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:52.254488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:52.254685 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:52.254612    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:52.575388 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:52.575224    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="d15a6d639e775fd76c2e3ec944d5bbabfe496fab8999bf6b735201a16d791a41" exitCode=0
Oct 31 07:28:52.575388 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:52.575265    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"d15a6d639e775fd76c2e3ec944d5bbabfe496fab8999bf6b735201a16d791a41"}
Oct 31 07:28:52.595545 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:52.595465    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-89wm7" podStartSLOduration=5.002186275 podStartE2EDuration="13.595445208s" totalImagesPullingTime="8.593258933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:51.596848034 +0000 UTC m=+12.787145063" watchObservedRunningTime="2026-10-31 07:28:52.595445208 +0000 UTC m=+13.785742231"
Oct 31 07:28:53.254887 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:53.254853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:53.255081 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:53.254904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:53.255081 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:53.254999    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:53.255409 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:53.255371    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:54.255405 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:54.255184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:54.255851 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:54.255546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:54.582348 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:54.582306    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerStarted","Data":"671d4bf89283a581ccf3a706fb6fc9e1d82abbaf34e56cb8d6c5a9dcc75bcdd2"}
Oct 31 07:28:55.255270 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.255241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:55.255392 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:55.255371    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:55.255433 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.255407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:55.255738 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:55.255479    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:55.587236 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.587012    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"a66d09d2c0e6f84cc3f832bd5676af717eb3ee8811c1df1cdd550915fed6a695"}
Oct 31 07:28:55.587364 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.587319    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"c2c2a6fc42b14bb113e3cf0eb1975737c3e83a3a1cc02208e8b2acbab1791b24"}
Oct 31 07:28:55.587408 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.587370    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"143e1b4dcae43a839025b96eb270fef8149898439c94977d2a62a3717a3915cf"}
Oct 31 07:28:55.587452 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.587411    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"1d5311e282260eb925261133dff62819b1446f1620af81611542d076bd3cf023"}
Oct 31 07:28:55.590241 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.590217    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="671d4bf89283a581ccf3a706fb6fc9e1d82abbaf34e56cb8d6c5a9dcc75bcdd2" exitCode=0
Oct 31 07:28:55.590325 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.590265    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"671d4bf89283a581ccf3a706fb6fc9e1d82abbaf34e56cb8d6c5a9dcc75bcdd2"}
Oct 31 07:28:55.939556 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:55.939495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:55.939735 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:55.939647    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:55.939735 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:55.939710    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret podName:bb9197ad-22d3-434a-9575-e60c63fe330a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:11.93969513 +0000 UTC m=+33.129992130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret") pod "global-pull-secret-syncer-xgqwg" (UID: "bb9197ad-22d3-434a-9575-e60c63fe330a") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:56.040865 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.040786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:56.040865 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.040827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.040929    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.040948    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.040969    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.040976    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:12.040963156 +0000 UTC m=+33.231260156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.040978    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4xnlt for pod openshift-network-diagnostics/network-check-target-56cfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:56.041033 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.041019    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt podName:8fbc61ab-9376-4c34-9452-443636430232 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:12.041008074 +0000 UTC m=+33.231305074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4xnlt" (UniqueName: "kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt") pod "network-check-target-56cfh" (UID: "8fbc61ab-9376-4c34-9452-443636430232") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:56.255099 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.255062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:56.255261 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:56.255190    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:56.595439 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.595406    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"004c1cd776b5b9fc53d9b020b0b803da143f7f2658473a666d08de686a0245a7"}
Oct 31 07:28:56.595967 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.595483    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"6eb7a2ccde965c88a8806e4fc3f69e54d03fa0e45e28efbc551c93e290ce173a"}
Oct 31 07:28:56.598284 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.598260    2582 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="afcfdb409cb5fd5241a58fa8106ee90105424f522274c890af937964c3dbb04a" exitCode=0
Oct 31 07:28:56.598401 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:56.598304    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerDied","Data":"afcfdb409cb5fd5241a58fa8106ee90105424f522274c890af937964c3dbb04a"}
Oct 31 07:28:57.254718 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:57.254680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:57.254718 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:57.254706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:57.254934 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:57.254793    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:28:57.254934 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:57.254904    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:57.603614 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:57.603535    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gxcql" event={"ID":"a1503777-02b6-4287-86f7-e1b7094397f2","Type":"ContainerStarted","Data":"c9dd5c2570a3d89d7322435c7e02440af97e1104f9ddf4c420f43ea7908ec3bc"}
Oct 31 07:28:57.624840 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:57.624795    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gxcql" podStartSLOduration=7.646204605 podStartE2EDuration="18.624779691s" totalImagesPullingTime="10.978575086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:57.624382535 +0000 UTC m=+18.814679554" watchObservedRunningTime="2026-10-31 07:28:57.624779691 +0000 UTC m=+18.815076712"
Oct 31 07:28:58.255011 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:58.254924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:28:58.255143 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:58.255032    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:28:58.608499 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:58.608417    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"29af3138a6a48bddb0d64dce2eb9246ffd95fc0fcea1c0086536de7b0b746549"}
Oct 31 07:28:59.255058 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:59.255027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:28:59.255198 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:59.255104    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:28:59.255313 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:28:59.255209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:28:59.255349 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:28:59.255323    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:00.254780 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.254589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:00.255311 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:00.254872    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:29:00.406082 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.406053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:00.556830 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.556706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:00.616737 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.616707    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" event={"ID":"2484cbbd-3954-4ee8-a0d2-ca0106675dae","Type":"ContainerStarted","Data":"302cc8880e52d04e1792f4f1c530dd7661814957cf3f7c6b144a6e94f3fdc6c1"}
Oct 31 07:29:00.617010 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.616982    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:00.632653 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.632630    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:00.643495 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:00.643455    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" podStartSLOduration=7.055966599 podStartE2EDuration="21.643443722s" totalImagesPullingTime="14.587477123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:00.643143412 +0000 UTC m=+21.833440434" watchObservedRunningTime="2026-10-31 07:29:00.643443722 +0000 UTC m=+21.833740744"
Oct 31 07:29:01.257561 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.257147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:01.257561 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:01.257288    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:29:01.258066 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.257743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:01.258066 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:01.257855    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:01.619195 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.619133    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:01.619647 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.619621    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:01.636584 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.636554    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:01.737272 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.737236    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xgqwg"]
Oct 31 07:29:01.737443 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.737365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:01.737504 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:01.737480    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:29:01.738898 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.738873    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56cfh"]
Oct 31 07:29:01.739081 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.738983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:01.739145 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:01.739083    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:29:01.739540 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.739502    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6glq2"]
Oct 31 07:29:01.739625 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:01.739612    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:01.739729 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:01.739711    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:02.621547 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:02.621503    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:03.146021 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.145991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.255180 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.255149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:03.255363 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.255147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:03.255363 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:03.255268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:29:03.255363 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:03.255331    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:03.255363 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.255163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:03.255547 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:03.255405    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:29:03.279853 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.279819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.622953 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:03.622925    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:04.200378 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:04.200337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:04.299562 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:04.299474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:05.254403 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.254374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:05.254403 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.254399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:05.254881 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:05.254548    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:29:05.254881 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.254586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:05.254881 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:05.254665    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:05.254881 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:05.254718    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:29:05.720709 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.720676    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:05.720880 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.720869    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:05.731334 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.731294    2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" podUID="2484cbbd-3954-4ee8-a0d2-ca0106675dae" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 07:29:05.739718 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:05.739686    2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp" podUID="2484cbbd-3954-4ee8-a0d2-ca0106675dae" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 07:29:06.697916 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:06.697883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:06.802023 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:06.801991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:06.971769 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:06.971737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:07.123844 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.123817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:07.245820 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.245741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:07.254680 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.254655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:07.254801 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.254662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:07.254801 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.254787    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glq2" podUID="41668c10-19d0-4d52-b138-81c2dea94d7f"
Oct 31 07:29:07.254871 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.254666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:07.254907 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.254820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgqwg" podUID="bb9197ad-22d3-434a-9575-e60c63fe330a"
Oct 31 07:29:07.254945 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.254903    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56cfh" podUID="8fbc61ab-9376-4c34-9452-443636430232"
Oct 31 07:29:07.582318 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.582240    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeReady"
Oct 31 07:29:07.582450 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.582354    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 07:29:07.610296 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.610266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:07.661001 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.660970    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x2pmn"]
Oct 31 07:29:07.663406 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.663382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.667089 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.666921    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wd96w"]
Oct 31 07:29:07.669882 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.669860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.673169 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.673150    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gd7dx\""
Oct 31 07:29:07.677295 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.677276    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:29:07.678285 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.678263    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x2pmn"]
Oct 31 07:29:07.683966 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.683943    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:07.686871 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.684384    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.686871 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.684659    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.686871 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.684693    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.686871 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.684734    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.689339 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.689305    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wd96w"]
Oct 31 07:29:07.690015 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.689995    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-llxtn"]
Oct 31 07:29:07.692770 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.692754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.705891 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.705866    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-llxtn"]
Oct 31 07:29:07.721615 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.721584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.721772 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.721635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrt7n\" (UniqueName: \"kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.773478 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.773442    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:07.773478 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.773447    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:29:07.773748 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.773735    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frjtn\""
Oct 31 07:29:07.773813 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.773801    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hfwqd\""
Oct 31 07:29:07.782697 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.782681    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:29:07.822933 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.822907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.823038 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.822937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xq95\" (UniqueName: \"kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.823038 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.822967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.823107 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.823107 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.823166 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4dh\" (UniqueName: \"kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.823252 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt7n\" (UniqueName: \"kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.823288 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.823319 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.823367 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.823418 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.823407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.827010 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.826991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.831416 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.831394    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt7n\" (UniqueName: \"kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:07.924602 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.924602 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.924602 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.924602 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xq95\" (UniqueName: \"kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.924683    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.924751    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.424734119 +0000 UTC m=+29.615031143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4dh\" (UniqueName: \"kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.924896 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.924921    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.925007    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.925024    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.925048    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.425035376 +0000 UTC m=+29.615332396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.925050    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.925183 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:07.925074    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls podName:2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.425060622 +0000 UTC m=+29.615357623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls") pod "dns-default-llxtn" (UID: "2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.925424 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.925403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.943331 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.943295    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4dh\" (UniqueName: \"kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:07.943451 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.943302    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xq95\" (UniqueName: \"kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:07.976370 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:07.976332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:08.167128 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.167099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.265453 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.265219    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x2pmn"]
Oct 31 07:29:08.270164 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:29:08.270131    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb4776c_2843_4b41_ab1e_1e65fa89ff36.slice/crio-8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f WatchSource:0}: Error finding container 8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f: Status 404 returned error can't find the container with id 8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f
Oct 31 07:29:08.293089 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.293055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.421412 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.421375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.429474 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.429448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:08.429575 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.429523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:08.429575 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.429557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:08.429682 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429594    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:08.429682 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429662    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:08.429682 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429680    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.429813 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429667    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.429647371 +0000 UTC m=+30.619944395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:08.429813 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429730    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.429716758 +0000 UTC m=+30.620013760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:08.429813 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:08.429750    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls podName:2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.429735738 +0000 UTC m=+30.620032738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls") pod "dns-default-llxtn" (UID: "2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.635682 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:08.635596    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x2pmn" event={"ID":"6eb4776c-2843-4b41-ab1e-1e65fa89ff36","Type":"ContainerStarted","Data":"8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f"}
Oct 31 07:29:09.256196 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.256159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:09.256686 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.256201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:09.256686 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.256529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:09.260253 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.260223    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:29:09.260407 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.260386    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:29:09.261247 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.261224    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:29:09.261800 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.261602    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ht7pz\""
Oct 31 07:29:09.261800 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.261647    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:29:09.261800 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.261659    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2shdh\""
Oct 31 07:29:09.261800 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.261665    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xr95\""
Oct 31 07:29:09.439074 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.439029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:09.439264 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439127    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:09.439264 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439194    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:11.43917508 +0000 UTC m=+32.629472086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:09.439264 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.439257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:09.439390 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:09.439304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:09.439390 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439341    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:09.439808 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439772    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls podName:2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:11.439751785 +0000 UTC m=+32.630048799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls") pod "dns-default-llxtn" (UID: "2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:09.439922 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439830    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:09.439922 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:09.439893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:11.439876013 +0000 UTC m=+32.630173024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:10.976053 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:10.976020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.455135 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.455095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:11.455292 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.455157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:11.455292 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.455200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:11.455292 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455260    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:11.455418 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455305    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:11.455418 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455328    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls podName:2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:15.455308078 +0000 UTC m=+36.645605081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls") pod "dns-default-llxtn" (UID: "2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:11.455418 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455354    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:15.455342532 +0000 UTC m=+36.645639533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:11.455418 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455305    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:11.455418 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:11.455382    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:15.455375558 +0000 UTC m=+36.645672558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:11.643061 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.643023    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x2pmn" event={"ID":"6eb4776c-2843-4b41-ab1e-1e65fa89ff36","Type":"ContainerStarted","Data":"96bf5d1895946ca5d7116a7152727eb8d5865fcdf52c00bcbeafac8bbf48c3e3"}
Oct 31 07:29:11.658236 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.658182    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x2pmn" podStartSLOduration=1.846416856 podStartE2EDuration="4.658167855s" totalImagesPullingTime="2.811750999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:11.657314977 +0000 UTC m=+32.847612007" watchObservedRunningTime="2026-10-31 07:29:11.658167855 +0000 UTC m=+32.848464876"
Oct 31 07:29:11.958839 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.958806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:11.959120 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.959094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.961154 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:11.961134    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:12.044101 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.044070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:12.059419 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.059381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:12.059419 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.059424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:12.059628 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:12.059532    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:12.059628 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:12.059585    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs podName:41668c10-19d0-4d52-b138-81c2dea94d7f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:44.0595709 +0000 UTC m=+65.249867899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs") pod "network-metrics-daemon-6glq2" (UID: "41668c10-19d0-4d52-b138-81c2dea94d7f") : secret "metrics-daemon-secret" not found
Oct 31 07:29:12.062145 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.062121    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:12.300073 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.300031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:12.326065 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.326030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.330237 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.330213    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xgqwg"]
Oct 31 07:29:12.333227 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:29:12.333193    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9197ad_22d3_434a_9575_e60c63fe330a.slice/crio-adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276 WatchSource:0}: Error finding container adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276: Status 404 returned error can't find the container with id adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276
Oct 31 07:29:12.435935 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.435902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.548294 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.548262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.621885 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.621808    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56cfh"]
Oct 31 07:29:12.625085 ip-10-0-137-69 kubenswrapper[2582]: W1031 07:29:12.625054    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbc61ab_9376_4c34_9452_443636430232.slice/crio-fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3 WatchSource:0}: Error finding container fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3: Status 404 returned error can't find the container with id fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3
Oct 31 07:29:12.646260 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.646222    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56cfh" event={"ID":"8fbc61ab-9376-4c34-9452-443636430232","Type":"ContainerStarted","Data":"fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3"}
Oct 31 07:29:12.647420 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.647396    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xgqwg" event={"ID":"bb9197ad-22d3-434a-9575-e60c63fe330a","Type":"ContainerStarted","Data":"adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276"}
Oct 31 07:29:12.960030 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:12.959954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:13.600841 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:13.600797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:13.767452 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:13.767393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:13.872669 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:13.872586    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xg255_42302c13-04fb-415c-affb-f652ff0be51c/dns-node-resolver/0.log"
Oct 31 07:29:13.958422 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:13.958374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.368466 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:14.368433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.549848 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:14.549812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.674719 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:14.674649    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkwsp_b89ab622-9b1b-4954-924b-b75da49f162f/node-ca/0.log"
Oct 31 07:29:15.168458 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.168420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.282486 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.282453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.379899 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.379864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.486567 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.486522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:15.486774 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.486588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:15.486774 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.486627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:15.486774 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:15.486642    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:15.486774 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:15.486718    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls podName:2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:23.486697684 +0000 UTC m=+44.676994700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls") pod "dns-default-llxtn" (UID: "2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:15.486774 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:15.486738    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:15.487022 ip-10-0-137-69 kubenswrapper[2582]: E1031 07:29:15.486793    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:23.486777499 +0000 UTC m=+44.677074504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:15.487225 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.487205    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:15.672656 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.672628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x2pmn_6eb4776c-2843-4b41-ab1e-1e65fa89ff36/serve-healthcheck-canary/0.log"
Oct 31 07:29:15.761571 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.761472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.975369 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:15.975338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.358694 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:16.358605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.557566 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:16.557527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.659572 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:16.659478    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56cfh" event={"ID":"8fbc61ab-9376-4c34-9452-443636430232","Type":"ContainerStarted","Data":"9dc4e1355da48cad6c03191a38f65589903ba19bfa0a8f45d7d247f5bc4e9d08"}
Oct 31 07:29:16.659733 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:16.659635    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:16.674598 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:16.674548    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-56cfh" podStartSLOduration=34.244534989 podStartE2EDuration="37.674529317s" totalImagesPullingTime="3.429994328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:16.673825111 +0000 UTC m=+37.864122134" watchObservedRunningTime="2026-10-31 07:29:16.674529317 +0000 UTC m=+37.864826342"
Oct 31 07:29:17.164865 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:17.164831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:17.441394 ip-10-0-137-69 kubenswrapper[2582]: I1031 07:29:17.441317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:17.525822 ip-10-0-137-69 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 07:29:17.533068 ip-10-0-137-69 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 07:29:17.533234 ip-10-0-137-69 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 07:29:17.533286 ip-10-0-137-69 systemd[1]: kubelet.service: Consumed 2.311s CPU time.
Oct 31 07:29:17.544220 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:29:17.625520 ip-10-0-137-69 kubenswrapper[4371]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:17.625520 ip-10-0-137-69 kubenswrapper[4371]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:29:17.625520 ip-10-0-137-69 kubenswrapper[4371]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:17.626137 ip-10-0-137-69 kubenswrapper[4371]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:17.627728 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627643    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.627728 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627729    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627733    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627736    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627739    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627742    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627745    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627748    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627751    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627753    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627756    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627759    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627761    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627764    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627766    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627769    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627771    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627774    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627776    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627779    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.627808 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627781    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627784    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627787    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627789    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627792    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627795    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627806    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627809    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627812    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627814    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627818    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627821    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627824    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627827    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627830    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627833    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627835    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627838    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627841    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627843    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.628290 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627846    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627849    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627851    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627854    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627856    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627859    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627862    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627865    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627868    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627870    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627873    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627876    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627879    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627882    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627885    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627888    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627892    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627897    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627900    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.628812 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627906    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627909    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627911    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627914    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627916    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627919    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627922    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627924    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627927    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627930    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627932    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627935    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627937    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627940    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627942    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627945    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627948    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627950    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627953    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627956    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.629299 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627958    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627961    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627963    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627966    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627969    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627971    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627974    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627978    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627982    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627985    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627989    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627992    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627995    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.627998    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628001    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628004    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628007    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628009    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628012    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628014    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.629820 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628017    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628020    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628023    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628026    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628029    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628031    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.628034    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629065    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629073    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629076    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629079    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629083    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629085    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629088    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629091    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629094    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629096    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629099    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629102    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629105    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.630333 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629108    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629111    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629113    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629116    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629118    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629121    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629124    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629127    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629130    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629132    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629135    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629137    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629140    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629142    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629146    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629148    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629151    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629154    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629156    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629159    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.630851 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629162    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629164    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629167    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629169    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629172    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629174    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629177    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629180    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629182    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629186    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629189    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629191    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629194    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629196    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629199    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629201    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629204    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629206    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629209    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.631349 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629212    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629215    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629217    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629220    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629223    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629225    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629228    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629230    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629233    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629237    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629240    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629244    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629247    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629250    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629253    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629256    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629258    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629261    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629263    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.631875 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629266    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629270    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629274    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629277    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629280    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629282    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629285    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629287    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629290    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629293    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629295    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629298    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629301    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629303    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629307    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629309    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629313    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629316    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629318    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629321    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.632346 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629323    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629326    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629329    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629331    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629334    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629336    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629339    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629342    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629366    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629371    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629373    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629376    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629379    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629382    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629388    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629930    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629945    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629950    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629954    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629957    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.632865 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629961    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629963    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629966    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629969    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629971    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629974    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629977    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629980    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629984    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629987    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629989    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629992    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629995    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.629998    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630001    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630003    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630006    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630009    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630012    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630014    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.633359 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630017    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630019    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630022    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630024    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630027    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630029    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630033    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630035    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630038    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630041    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630043    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630046    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630048    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630051    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630053    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630058    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630062    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630065    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630069    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.633884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630074    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630077    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630080    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630083    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630085    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630088    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630091    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630093    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630096    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630098    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630101    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630103    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630106    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630109    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630112    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630114    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630117    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630119    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630122    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630125    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.634411 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630128    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630130    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630133    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630136    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630139    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630141    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630144    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630146    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630149    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630151    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630154    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630157    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630159    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630162    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630165    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630167    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630170    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630173    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630176    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.635645 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630178    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630181    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630183    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630186    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630189    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630191    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630193    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630196    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630198    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630201    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630203    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630206    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630208    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630211    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630214    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630216    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630219    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630221    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630224    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630228    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.636295 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630230    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630233    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.630235    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630345    4371 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630358    4371 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630365    4371 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630370    4371 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630375    4371 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630379    4371 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630384    4371 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630388    4371 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630391    4371 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630395    4371 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630398    4371 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630402    4371 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630405    4371 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630408    4371 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630411    4371 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630414    4371 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630417    4371 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630423    4371 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630427    4371 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630430    4371 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630433    4371 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:29:17.636819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630436    4371 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630440    4371 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630444    4371 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630447    4371 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630450    4371 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630453    4371 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630457    4371 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630460    4371 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630463    4371 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630466    4371 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630469    4371 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630474    4371 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630477    4371 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630481    4371 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630484    4371 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630487    4371 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630491    4371 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630496    4371 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630499    4371 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630502    4371 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630519    4371 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630525    4371 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630532    4371 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630535    4371 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630539    4371 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:29:17.637408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630542    4371 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630545    4371 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630548    4371 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630555    4371 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630559    4371 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630562    4371 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630565    4371 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630568    4371 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630573    4371 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630576    4371 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630580    4371 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630583    4371 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630586    4371 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630589    4371 flags.go:64] FLAG: --help="false"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630592    4371 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630597    4371 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630601    4371 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630604    4371 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630607    4371 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630611    4371 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630614    4371 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630617    4371 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630620    4371 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:29:17.638096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630623    4371 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630627    4371 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630630    4371 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630633    4371 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630636    4371 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630639    4371 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630643    4371 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630646    4371 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630648    4371 flags.go:64] FLAG: --lock-file=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630651    4371 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630655    4371 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630658    4371 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630664    4371 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630667    4371 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630670    4371 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630673    4371 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630676    4371 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630680    4371 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630683    4371 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630686    4371 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630691    4371 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630694    4371 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630698    4371 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630701    4371 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630706    4371 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:29:17.638686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630709    4371 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630712    4371 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630715    4371 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630718    4371 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630721    4371 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630729    4371 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630732    4371 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630735    4371 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630739    4371 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630743    4371 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630746    4371 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630749    4371 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630752    4371 flags.go:64] FLAG: --port="10250"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630755    4371 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630758    4371 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c5d28f9418ef944c"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630762    4371 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630765    4371 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630768    4371 flags.go:64] FLAG: --register-node="true"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630771    4371 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630776    4371 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630779    4371 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630782    4371 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630785    4371 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630789    4371 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630792    4371 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630795    4371 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:29:17.639339 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630798    4371 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630801    4371 flags.go:64] FLAG: --runonce="false"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630805    4371 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630808    4371 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630811    4371 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630815    4371 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630819    4371 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630822    4371 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630825    4371 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630828    4371 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630831    4371 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630834    4371 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630838    4371 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630841    4371 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630844    4371 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630847    4371 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630850    4371 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630856    4371 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630859    4371 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630862    4371 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630866    4371 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630869    4371 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630872    4371 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630875    4371 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630878    4371 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:29:17.640175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630882    4371 flags.go:64] FLAG: --v="2"
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630887    4371 flags.go:64] FLAG: --version="false"
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630891    4371 flags.go:64] FLAG: --vmodule=""
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630895    4371 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.630898    4371 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631037    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631041    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631044    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631046    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631049    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631051    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631054    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631058    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631061    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631064    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631067    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631069    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631073    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631077    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631080    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.640798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631083    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631086    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631089    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631091    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631095    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631097    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631100    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631103    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631105    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631108    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631111    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631114    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631118    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631121    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631123    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631126    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631128    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631131    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631134    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631136    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.641451 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631139    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631142    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631144    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631147    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631150    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631153    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631155    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631158    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631160    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631163    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631166    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631168    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631171    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631173    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631176    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631179    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631182    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631185    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631187    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.642116 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631190    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631193    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631196    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631198    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631201    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631205    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631207    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631210    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631212    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631215    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631218    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631220    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631223    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631226    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631228    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631231    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631233    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631237    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631240    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631242    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.642707 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631245    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631247    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631250    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631252    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631255    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631258    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631260    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631263    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631266    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631270    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631274    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631278    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631282    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631285    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631289    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631291    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631294    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631299    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631301    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.643252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631304    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631307    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631309    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631312    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631315    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631317    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631320    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631322    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631325    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631327    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631333    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631336    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631339    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.631365    4371 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631400    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631404    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.644153 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631408    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631411    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631414    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631417    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631419    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631422    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631424    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631427    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631430    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631433    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631436    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631439    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631441    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631444    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631447    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631451    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631453    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631456    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631458    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631461    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.644781 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631464    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631466    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631469    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631471    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631474    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631477    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631480    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631484    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631488    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631491    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631494    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631497    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631500    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631503    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631519    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631522    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631525    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631528    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631531    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.645454 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631534    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631536    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631539    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631542    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631545    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631548    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631551    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631553    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631557    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631560    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631562    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631565    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631568    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631570    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631573    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631575    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631578    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631581    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631583    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631587    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.645956 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631590    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631594    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631597    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631599    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631603    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631607    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631611    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631614    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631616    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631619    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631622    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631625    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631628    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631630    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631633    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631636    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631639    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631642    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631644    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.646462 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631647    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631651    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631654    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631656    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631659    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631662    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631664    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631667    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631669    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631672    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631674    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631677    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631681    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631684    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631686    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631689    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631691    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631694    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631696    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.646946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631699    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631702    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631705    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631707    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631710    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631713    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.631715    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.636631    4371 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.636648    4371 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.636668    4371 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.636673    4371 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636742    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636747    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636750    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636754    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636757    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636760    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636763    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636766    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.647449 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636768    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636771    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636774    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636776    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636779    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636782    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636785    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636788    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636791    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636794    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636796    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636799    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636802    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636805    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636807    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636810    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636813    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636815    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636818    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.647953 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636821    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636823    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636826    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636830    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636834    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636837    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636841    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636844    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636846    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636849    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636852    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636854    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636857    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636860    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636863    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636865    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636868    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636871    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636874    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.648433 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636877    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636879    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636882    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636884    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636887    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636890    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636892    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636895    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636897    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636899    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636903    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636905    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636908    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636911    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636913    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636916    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636919    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636921    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636924    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636927    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.649003 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636930    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636932    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636935    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636937    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636940    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636943    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636946    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636948    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636951    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636953    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636956    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636959    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636961    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636965    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636967    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636970    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636973    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636976    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636978    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.649485 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636981    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636984    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636988    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636991    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636994    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.636997    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637000    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637002    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637005    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637007    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637010    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637013    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637015    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637035    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637038    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637040    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637043    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637046    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637049    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637051    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.650048 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637054    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.637085    4371 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637234    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637239    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637242    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637245    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637247    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637251    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637253    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637256    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637259    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637261    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637264    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637267    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637270    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637275    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.650552 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637278    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637282    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637286    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637289    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637292    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637294    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637297    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637299    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637303    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637305    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637308    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637311    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637316    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637319    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637321    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637324    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637326    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637329    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637332    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.650946 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637335    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637337    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637341    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637345    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637348    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637352    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637355    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637358    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637360    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637363    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637366    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637368    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637372    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637375    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637378    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637381    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637383    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637386    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637388    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637391    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.651395 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637393    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637396    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637398    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637401    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637403    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637407    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637410    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637412    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637415    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637418    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637421    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637424    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637427    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637430    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637432    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637435    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637438    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637440    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637444    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.651897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637446    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637449    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637451    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637454    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637456    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637460    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637463    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637466    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637468    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637471    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637473    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637476    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637478    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637481    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637484    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637487    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637489    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637492    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637495    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637497    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.652453 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637500    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637503    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637517    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637520    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637523    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637525    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637528    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637530    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637533    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637536    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637538    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637541    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637544    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637547    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.637571    4371 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637604    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:17.653252 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637609    4371 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637612    4371 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637616    4371 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637619    4371 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637622    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637625    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637627    4371 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637630    4371 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637632    4371 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637635    4371 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637638    4371 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637641    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637643    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637647    4371 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637650    4371 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637652    4371 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637655    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637658    4371 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637661    4371 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:17.653807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637664    4371 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637666    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637669    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637672    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637674    4371 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637677    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637679    4371 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637682    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637685    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637687    4371 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637690    4371 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637693    4371 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637696    4371 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637698    4371 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637701    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637705    4371 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637708    4371 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637710    4371 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637713    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637715    4371 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:17.654289 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637718    4371 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637720    4371 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637723    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637725    4371 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637728    4371 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637731    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637734    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637736    4371 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637739    4371 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637741    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637744    4371 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637746    4371 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637749    4371 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637751    4371 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637754    4371 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637757    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637759    4371 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637762    4371 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637765    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:17.654798 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637767    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637770    4371 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637772    4371 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637775    4371 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637778    4371 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637781    4371 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637783    4371 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637786    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637791    4371 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637794    4371 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637796    4371 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637799    4371 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637802    4371 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637805    4371 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637808    4371 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637811    4371 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637815    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637818    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637821    4371 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:17.655332 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637823    4371 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637826    4371 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637828    4371 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637831    4371 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637834    4371 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637836    4371 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637839    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637841    4371 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637844    4371 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637846    4371 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637849    4371 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637851    4371 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637854    4371 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637857    4371 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637860    4371 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637863    4371 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637867    4371 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637870    4371 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637874    4371 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637877    4371 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:17.656218 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637880    4371 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637884    4371 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637887    4371 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637889    4371 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637892    4371 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637895    4371 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637897    4371 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:17.637900    4371 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.637987    4371 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.638985    4371 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639047    4371 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639215    4371 server.go:1014] "Starting client certificate rotation"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639419    4371 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639472    4371 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:39 +0000 UTC" deadline="2028-04-14 03:52:32.868819274 +0000 UTC"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639557    4371 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12740h23m15.229265747s"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.639840    4371 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:17.656884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.641048    4371 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:17.657314 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.642727    4371 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:29:17.657314 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.644678    4371 log.go:25] "Validated CRI v1 image API"
Oct 31 07:29:17.657314 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.645403    4371 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:29:17.657314 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.646918    4371 fs.go:135] Filesystem UUIDs: map[330852c8-a011-423b-96b6-f2566b82091d:/dev/nvme0n1p3 557a13b1-392c-495d-9043-042406b724f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 07:29:17.662812 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:29:17.663886 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.646940    4371 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7/userdata/shm major:0 minor:57 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd/userdata/shm major:0 minor:53 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~projected/kube-api-access-tpflk:{mountpoint:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~projected/kube-api-access-tpflk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2784ec-df9b-4a2e-8dc6-4af7d170341a/volumes/kubernetes.io~projected/kube-api-access-5xq95:{mountpoint:/var/lib/kubelet/pods/0b2784ec-df9b-4a2e-8dc6-4af7d170341a/volumes/kubernetes.io~projected/kube-api-access-5xq95 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~projected/kube-api-access-8dqtm:{mountpoint:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~projected/kube-api-access-8dqtm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4/volumes/kubernetes.io~projected/kube-api-access-7c4dh:{mountpoint:/var/lib/kubelet/pods/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4/volumes/kubernetes.io~projected/kube-api-access-7c4dh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41668c10-19d0-4d52-b138-81c2dea94d7f/volumes/kubernetes.io~projected/kube-api-access-nstks:{mountpoint:/var/lib/kubelet/pods/41668c10-19d0-4d52-b138-81c2dea94d7f/volumes/kubernetes.io~projected/kube-api-access-nstks major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42302c13-04fb-415c-affb-f652ff0be51c/volumes/kubernetes.io~projected/kube-api-access-z2s4x:{mountpoint:/var/lib/kubelet/pods/42302c13-04fb-415c-affb-f652ff0be51c/volumes/kubernetes.io~projected/kube-api-access-z2s4x major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49be424d-4007-47e0-97e2-e8b63e9af6eb/volumes/kubernetes.io~projected/kube-api-access-bjnhv:{mountpoint:/var/lib/kubelet/pods/49be424d-4007-47e0-97e2-e8b63e9af6eb/volumes/kubernetes.io~projected/kube-api-access-bjnhv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b12ceb4-d721-4fe8-9df6-644b2023cd74/volumes/kubernetes.io~projected/kube-api-access-w8g9w:{mountpoint:/var/lib/kubelet/pods/4b12ceb4-d721-4fe8-9df6-644b2023cd74/volumes/kubernetes.io~projected/kube-api-access-w8g9w major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67890a7c-54d4-4af5-a01c-fee014b737a8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/67890a7c-54d4-4af5-a01c-fee014b737a8/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~projected/kube-api-access-hrt7n:{mountpoint:/var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~projected/kube-api-access-hrt7n major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fbc61ab-9376-4c34-9452-443636430232/volumes/kubernetes.io~projected/kube-api-access-4xnlt:{mountpoint:/var/lib/kubelet/pods/8fbc61ab-9376-4c34-9452-443636430232/volumes/kubernetes.io~projected/kube-api-access-4xnlt major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1503777-02b6-4287-86f7-e1b7094397f2/volumes/kubernetes.io~projected/kube-api-access-gmcgd:{mountpoint:/var/lib/kubelet/pods/a1503777-02b6-4287-86f7-e1b7094397f2/volumes/kubernetes.io~projected/kube-api-access-gmcgd major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b89ab622-9b1b-4954-924b-b75da49f162f/volumes/kubernetes.io~projected/kube-api-access-nvvj7:{mountpoint:/var/lib/kubelet/pods/b89ab622-9b1b-4954-924b-b75da49f162f/volumes/kubernetes.io~projected/kube-api-access-nvvj7 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb9197ad-22d3-434a-9575-e60c63fe330a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bb9197ad-22d3-434a-9575-e60c63fe330a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccd6ee2e-4015-471d-9fe6-b9efc31594ae/volumes/kubernetes.io~projected/kube-api-access-rt9g5:{mountpoint:/var/lib/kubelet/pods/ccd6ee2e-4015-471d-9fe6-b9efc31594ae/volumes/kubernetes.io~projected/kube-api-access-rt9g5 major:0 minor:51 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5f20f44622c652db1af7c2b031d5ab32408b9bb108b42bb2e3a64a178eed6dc4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f7ae5a055bb42bb1eb6a65b73e21a5e9f7bc0c8d9fd665b76a8744bd7456e3d7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8385a2a263719d2b45947af57a3551bec63a4238cf54f20dc936dad22fafe98e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0ac17d7233d470b2242de3850908758c995b1bda5e6f0547918d330781917fa3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1057b50562559cb736568bb3b577873c5bb42a08d285466ed1c3f031050792c7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f34717ca17914b091259debb03b31c61c70f58f386084749a0d2902700e7cb1b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ef919d6374862736ca0402badf3b4841c1759f53e398fecaedb27496bebe8d47/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2d63e5ea5312993968de465a8f66a7c342442c350abd3a06a64532fe6b44b42f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0d220fd0ba941b26b741c0319896933e37712e68539b1342fb93cf4128093e47/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/c12335d7a62ec7fecbc45ee4f59d2cd6d8af31684fef28a7a3784ad5f1a62f32/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/23d09273000ff922faf64f777ba1d3d0c18ca765787e1352fc985563f9099d21/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7c16839a54a1a19426dac7e902b9705eaed3d1fac2950b8cc92324fda17411e4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/086c37b345a2a575b7a5d51e8fbf092b00683b87946cbd76c6a3fce5d34ec4fd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/02de22d84e81d664bc889fba0b5aed9996333d0690cbea9385e3e95b41292088/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4bfc655635b883e59430aab8af2ce8e3cbf64bce34ee400a98285442813091a8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/34293821f5d2597ad94a499c6aacc99dffa7ae8628a71a9a1d0974377a849489/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8ad3f93dad4110c584a9937650c5721f5e165aa1a08b05af1f3b8f2ff70cb7e0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/00535414656aa54514f4b9074dd4419019edd9753b86e67d59f4d911ceca5e0c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/5cc5bccb0b0dd0c6444f4a22eb52d8ac93baa172902ad0ccbfdd436331a268bf/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/7ca1a9993251e4317ebebd410d89bd3ee73a670b47c3e7ec87260686c41144ec/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fe5eba650c00d80fb9d2ca33a8fcb93eefe2a56a027f6807932a26b896f61dba/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/3dae73e1506334ce85c3562f01b5e243df1cea41d7a55244d083083137fcf128/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/2da514fdd073156c56a6bdbf9e48572d16d3f0186afa5a22daf82df8f0d14949/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/06937fdd218e8c16677822e1fa4c064235404fa65eb505188a50948ffbc33863/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/9e07fcb666033e4e213b501b5e5a6ebcafc4c183acf4def2c0f20f665415ff7f/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/d08dc57adc032c0a105814a89ad0bc700fc93948be86f4522008f450735a374f/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/d0081bda997f63d84faaa22c14f27e50c6f9100470c977269bdc4380a1cc5c0d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/8eb55fcfbde3e77af7c5b8727d59c96a750ef8f7c3341a7096b48740f5f279aa/merged major:0 minor:55 fsType:overlay blockSize:0} overlay_0-59:{mountpoint:/var/lib/containers/storage/overlay/4488910e5642eddf492120ec7260257e28d5b67b7284d606066354baa36ce76a/merged major:0 minor:59 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fe6a59367ec8d640d3e7f4eed467eb30b9cca8b414144ff0a0b5627b9436f41f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/22d0fff5282b4e23c1d843146424244cd288339b6e2bc4354c057209b91d307a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f00b13802e73e0630609d950a75e9fbfdca75a9ebae4fabc35b08d326375151f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1e39c57bbc776371120bf2899a4748c2409defc6c8477c0483c29ca5ca7e0b79/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/cd71ea1b9631828d3aeafe33705914a40ed0cd342d5f127e400dd38df08ef281/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/01e2dac137ac58901df7d4427e7ac87d35997ca212c2ffe7dd029cc04e998008/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/95f8186e0bd9622b99d66cf85fc7ee2aa8840f0cd3a59988e3b17d3fa58c4158/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6031f6a888490d7389165aab419aaa58c6a1621eaf7b4b6eccdc9730d81731b2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.653853    4371 manager.go:217] Machine: {Timestamp:2026-10-31 07:29:17.652905392 +0000 UTC m=+0.069036056 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098505 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c559277247202126e6cdbb829b90f SystemUUID:ec2c5592-7724-7202-126e-6cdbb829b90f BootID:2d54593a-85b7-4f4b-b69e-aa4943c92b4e Filesystems:[{Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b89ab622-9b1b-4954-924b-b75da49f162f/volumes/kubernetes.io~projected/kube-api-access-nvvj7 DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2784ec-df9b-4a2e-8dc6-4af7d170341a/volumes/kubernetes.io~projected/kube-api-access-5xq95 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-59 DeviceMajor:0 DeviceMinor:59 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd/userdata/shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~projected/kube-api-access-tpflk DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a34bd53559b4ee867376307bcdea44b9e0f6e5887a1dcbd8be35277cb663fbd2/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/41668c10-19d0-4d52-b138-81c2dea94d7f/volumes/kubernetes.io~projected/kube-api-access-nstks DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~projected/kube-api-access-hrt7n DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42302c13-04fb-415c-affb-f652ff0be51c/volumes/kubernetes.io~projected/kube-api-access-z2s4x DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d08296fb511bb7720430da92b37be74a9666a099dd8bd23f6a0df0eccc78e40/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a7eaa89867514306ba8b9b99bd42ebffb1bcb687813c02fe740263a68522398/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4/volumes/kubernetes.io~projected/kube-api-access-7c4dh DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8fbc61ab-9376-4c34-9452-443636430232/volumes/kubernetes.io~projected/kube-api-access-4xnlt DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ccd6ee2e-4015-471d-9fe6-b9efc31594ae/volumes/kubernetes.io~projected/kube-api-access-rt9g5 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volumes/kubernetes.io~projected/kube-api-access-8dqtm DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67890a7c-54d4-4af5-a01c-fee014b737a8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/49be424d-4007-47e0-97e2-e8b63e9af6eb/volumes/kubernetes.io~projected/kube-api-access-bjnhv DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2484cbbd-3954-4ee8-a0d2-ca0106675dae/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8639ce151aa7aa8a6ebbc7c6f5c37a2399cbe4707c5836f1f0727bab0c9dcb6f/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4b12ceb4-d721-4fe8-9df6-644b2023cd74/volumes/kubernetes.io~projected/kube-api-access-w8g9w DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6eb4776c-2843-4b41-ab1e-1e65fa89ff36/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6042a6f3275b6a92f7bb1102ebd0b56be493688d035ae3d9e9e3e79ffaa84d95/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b20efa1631e2f37cc9aa948a26717c705da878915d4a2aaf0bdc1313032c560e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adb4a6a86cbcd3139bee2264bc0bb90f3cf6911c3500f90ef40a854132eb2276/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c53723de6c38cc8a2d2f01bea5fd46881f304cecba23bda0ae23308160348fe4/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bb9197ad-22d3-434a-9575-e60c63fe330a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc305083af80fb8d4ac7a404f55498d001001b4c656fd34efa35fb940ebd26b3/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad2e2d699b3779ef1c7dc7670928e4811a3014f791559ac498dbfc57a68c29ef/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1503777-02b6-4287-86f7-e1b7094397f2/volumes/kubernetes.io~projected/kube-api-access-gmcgd DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7/userdata/shm DeviceMajor:0 DeviceMinor:57 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dacce6eb6046cc9510c56aba51867424808764b0b5301bd3fffc2f23a21db01/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8639ce151aa7aa8 MacAddress:82:ed:1e:c5:11:3f Speed:10000 Mtu:8901} {Name:adb4a6a86cbcd31 MacAddress:16:86:2f:08:5e:84 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c2:e3:c9:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:bc:b6:37:91:5a Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c2:e3:c9:ef Speed:0 Mtu:9001} {Name:fc305083af80fb8 MacAddress:62:ca:52:1a:8b:9d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:ad:43:b0:c9:52 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:32:6b:55:db:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654134    4371 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654195    4371 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654473    4371 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654500    4371 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654652    4371 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654659    4371 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654670    4371 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654690    4371 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654794    4371 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.654960    4371 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655075    4371 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655114    4371 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655123    4371 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655133    4371 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655143    4371 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655152    4371 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655738    4371 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.655750    4371 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.656308    4371 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658105    4371 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658114    4371 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658212    4371 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658404    4371 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658430    4371 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658539    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658551    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658558    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658564    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658570    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658576    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658582    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658587    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658594    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658600    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658608    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658618    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658645    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.658651    4371 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.662232    4371 server.go:1263] "Started kubelet"
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.662293    4371 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.662418    4371 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.663072    4371 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:29:17.664139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.663434    4371 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:29:17.666308 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.665387    4371 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:29:17.668882 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.668585    4371 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:29:17.668974 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.668911    4371 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:29:17.668974 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.668927    4371 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:29:17.668974 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.668923    4371 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.us-east-2.compute.internal\" not found"
Oct 31 07:29:17.669162 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.669050    4371 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:29:17.669162 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.669128    4371 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:29:17.669321 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.669236    4371 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:40 +0000 UTC" deadline="2028-05-12 22:26:17.463014061 +0000 UTC"
Oct 31 07:29:17.669321 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.669269    4371 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13430h56m59.793748346s"
Oct 31 07:29:17.670282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.669854    4371 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:17.670282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.670212    4371 factory.go:55] Registering systemd factory
Oct 31 07:29:17.670282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.670229    4371 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:29:17.671147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.670947    4371 factory.go:153] Registering CRI-O factory
Oct 31 07:29:17.671147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.670968    4371 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:29:17.671147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.671027    4371 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:29:17.671147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.671048    4371 factory.go:103] Registering Raw factory
Oct 31 07:29:17.671147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.671064    4371 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:29:17.672245 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.672228    4371 manager.go:319] Starting recovery of all containers
Oct 31 07:29:17.672965 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.672884    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fbc61ab-9376-4c34-9452-443636430232" volumeName="kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.672967    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1503777-02b6-4287-86f7-e1b7094397f2" volumeName="kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.672982    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1503777-02b6-4287-86f7-e1b7094397f2" volumeName="kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.672994    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd6ee2e-4015-471d-9fe6-b9efc31594ae" volumeName="kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673006    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4" volumeName="kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673018    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67890a7c-54d4-4af5-a01c-fee014b737a8" volumeName="kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673029    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd6ee2e-4015-471d-9fe6-b9efc31594ae" volumeName="kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673040    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2784ec-df9b-4a2e-8dc6-4af7d170341a" volumeName="kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673055    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49be424d-4007-47e0-97e2-e8b63e9af6eb" volumeName="kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673069    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b12ceb4-d721-4fe8-9df6-644b2023cd74" volumeName="kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673080    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb4776c-2843-4b41-ab1e-1e65fa89ff36" volumeName="kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673092    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b89ab622-9b1b-4954-924b-b75da49f162f" volumeName="kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673129    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb4776c-2843-4b41-ab1e-1e65fa89ff36" volumeName="kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673155    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1503777-02b6-4287-86f7-e1b7094397f2" volumeName="kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:17.673159 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673166    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="073c9d08-d5d6-4a1b-b39a-fc5a2ab21664" volumeName="kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673175    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67890a7c-54d4-4af5-a01c-fee014b737a8" volumeName="kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673182    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="073c9d08-d5d6-4a1b-b39a-fc5a2ab21664" volumeName="kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673190    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="073c9d08-d5d6-4a1b-b39a-fc5a2ab21664" volumeName="kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673198    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2784ec-df9b-4a2e-8dc6-4af7d170341a" volumeName="kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673237    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42302c13-04fb-415c-affb-f652ff0be51c" volumeName="kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673246    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1503777-02b6-4287-86f7-e1b7094397f2" volumeName="kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673254    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2484cbbd-3954-4ee8-a0d2-ca0106675dae" volumeName="kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673262    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2484cbbd-3954-4ee8-a0d2-ca0106675dae" volumeName="kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673269    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4" volumeName="kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673276    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b89ab622-9b1b-4954-924b-b75da49f162f" volumeName="kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673283    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2484cbbd-3954-4ee8-a0d2-ca0106675dae" volumeName="kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673290    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41668c10-19d0-4d52-b138-81c2dea94d7f" volumeName="kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673298    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4" volumeName="kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673309    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2784ec-df9b-4a2e-8dc6-4af7d170341a" volumeName="kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673333    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2484cbbd-3954-4ee8-a0d2-ca0106675dae" volumeName="kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673359    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2484cbbd-3954-4ee8-a0d2-ca0106675dae" volumeName="kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673372    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42302c13-04fb-415c-affb-f652ff0be51c" volumeName="kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673384    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49be424d-4007-47e0-97e2-e8b63e9af6eb" volumeName="kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673398    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb9197ad-22d3-434a-9575-e60c63fe330a" volumeName="kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673410    4371 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccd6ee2e-4015-471d-9fe6-b9efc31594ae" volumeName="kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673495    4371 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673523    4371 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:29:17.673978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.673740    4371 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:17.675822 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.675801    4371 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:17.676740 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.676723    4371 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 07:29:17.697180 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.697097    4371 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:29:17.698361 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.698338    4371 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:29:17.698361 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.698365    4371 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:29:17.698583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.698384    4371 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:29:17.698583 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.698418    4371 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 07:29:17.702280 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.702256    4371 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:17.705396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.705370    4371 generic.go:356] "Generic (PLEG): container finished" podID="8ecc22959f6c5e0b1bb2a00c01e37014" containerID="d1c412f6db1732f1caa295a6ea5fc134eb43f4d37f822b8bd1485b7b6d02ba6b" exitCode=0
Oct 31 07:29:17.729223 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729189    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="afcfdb409cb5fd5241a58fa8106ee90105424f522274c890af937964c3dbb04a" exitCode=0
Oct 31 07:29:17.729223 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729218    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="671d4bf89283a581ccf3a706fb6fc9e1d82abbaf34e56cb8d6c5a9dcc75bcdd2" exitCode=0
Oct 31 07:29:17.729223 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729226    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="d15a6d639e775fd76c2e3ec944d5bbabfe496fab8999bf6b735201a16d791a41" exitCode=0
Oct 31 07:29:17.729438 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729234    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="94196f6e8c970ad088f7d8aea7136572934eb06d20d3ce198cec3e141bb84ce8" exitCode=0
Oct 31 07:29:17.729438 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729244    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="9c6e4eccadde3626134f41570940e8c3fc624a91636ab9ef45f654f8b2a7211b" exitCode=0
Oct 31 07:29:17.729438 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.729251    4371 generic.go:356] "Generic (PLEG): container finished" podID="a1503777-02b6-4287-86f7-e1b7094397f2" containerID="6f2fecc072c8e916caab1ad61a9f57769d833654d1920b39375be587c5a8e920" exitCode=0
Oct 31 07:29:17.739304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.739278    4371 manager.go:324] Recovery completed
Oct 31 07:29:17.752733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752711    4371 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:29:17.752733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752726    4371 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752746    4371 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752864    4371 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752876    4371 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752894    4371 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752899    4371 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 07:29:17.752902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752905    4371 policy_none.go:50] "Start"
Oct 31 07:29:17.753086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752914    4371 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:29:17.753086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.752922    4371 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:29:17.753086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.753054    4371 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 07:29:17.753086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.753062    4371 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 07:29:17.753086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.753068    4371 policy_none.go:44] "Start"
Oct 31 07:29:17.757043 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.757008    4371 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:29:17.759827 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.759811    4371 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:29:17.760393 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.760373    4371 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:29:17.760393 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760389    4371 server.go:87] "Starting device plugin registration server"
Oct 31 07:29:17.760670 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760656    4371 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:29:17.760736 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760668    4371 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:29:17.760787 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760765    4371 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:29:17.760903 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760871    4371 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:29:17.760961 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760913    4371 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:29:17.760961 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760920    4371 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:29:17.761057 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.760979    4371 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:29:17.760902209Z","UUID":"d81f691e-996d-4d29-a5cc-46cf77553db0","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:29:17.762212 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.762192    4371 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:29:17.762347 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.762327    4371 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:29:17.762432 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.762352    4371 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:29:17.799521 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.799461    4371 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"]
Oct 31 07:29:17.799980 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.799950    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerStarted","Data":"26bae6f15b69f418bfd0916b1ae63d8f5b130986025d926f4257e9fba3e5ddf6"}
Oct 31 07:29:17.799980 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.799988    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerDied","Data":"d1c412f6db1732f1caa295a6ea5fc134eb43f4d37f822b8bd1485b7b6d02ba6b"}
Oct 31 07:29:17.800129 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.800009    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"8ecc22959f6c5e0b1bb2a00c01e37014","Type":"ContainerStarted","Data":"49e8c6f45de3448442d55236112b613bfa78d5d59d5eca6c099b673cdda9fde7"}
Oct 31 07:29:17.800129 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.800027    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"7057eabb38a9074e4b218a9c4b8b8759","Type":"ContainerStarted","Data":"c378e6ec4b0c4eb27ba818ec45c34d8afb9a26cb6fd9688e5add5aa2ea040ac5"}
Oct 31 07:29:17.800129 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.800043    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal" event={"ID":"7057eabb38a9074e4b218a9c4b8b8759","Type":"ContainerStarted","Data":"77796dbef45e4b015d38cee38afa07627c4d315ebe3c2d4302d0529ad6970ffd"}
Oct 31 07:29:17.800299 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.800262    4371 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.800299 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.800279    4371 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.811899 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.811878    4371 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:17.812015 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.811960    4371 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.812401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.812385    4371 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:17.812455 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:17.812425    4371 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.861595 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.861563    4371 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:29:17.862861 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.862842    4371 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:29:17.862965 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.862873    4371 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:29:17.862965 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.862884    4371 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:29:17.862965 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.862915    4371 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.874540 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.874499    4371 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.874640 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.874583    4371 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.877324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.877303    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.877429 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.877330    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.877429 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.877368    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977758 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977725    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977755    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7057eabb38a9074e4b218a9c4b8b8759-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"7057eabb38a9074e4b218a9c4b8b8759\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977836    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977861    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977911    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:17.977970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:17.977914    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ecc22959f6c5e0b1bb2a00c01e37014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" (UID: \"8ecc22959f6c5e0b1bb2a00c01e37014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:18.656089 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.656053    4371 apiserver.go:52] "Watching apiserver"
Oct 31 07:29:18.668709 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.668674    4371 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:29:18.669100 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.669077    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6glq2","openshift-ingress-canary/ingress-canary-x2pmn","openshift-dns/node-resolver-xg255","openshift-dns/dns-default-llxtn","openshift-network-diagnostics/network-check-target-56cfh","openshift-ovn-kubernetes/ovnkube-node-lb4zp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd","kube-system/konnectivity-agent-5dvbr","openshift-multus/multus-additional-cni-plugins-gxcql","kube-system/global-pull-secret-syncer-xgqwg","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-clpj5","openshift-multus/multus-gt6sx","openshift-network-operator/iptables-alerter-89wm7","openshift-insights/insights-runtime-extractor-wd96w","openshift-image-registry/node-ca-kkwsp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"]
Oct 31 07:29:18.674771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.674745    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.676387 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.676332    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.677157 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.677138    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.678740 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.678714    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.678838 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.678790    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6w9r4\""
Oct 31 07:29:18.678838 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.678812    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9r8qc\""
Oct 31 07:29:18.679440 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.679417    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.680396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680377    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.680466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680394    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bn5lf\""
Oct 31 07:29:18.680466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680395    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:29:18.680466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680423    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:29:18.680466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680434    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.680466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680442    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:29:18.680875 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680861    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:29:18.680924 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680889    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ht7pz\""
Oct 31 07:29:18.680924 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680890    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2shdh\""
Oct 31 07:29:18.680994 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680940    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.680994 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680950    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.680994 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680975    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.680994 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.680992    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.681142 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681010    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681142 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681065    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681233 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681195    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcgd\" (UniqueName: \"kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681272 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681231    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.681311 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681268    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.681311 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681298    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681393 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681342    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681447    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681466 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681456    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681608 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681585    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1503777-02b6-4287-86f7-e1b7094397f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.681750 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.681737    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb9197ad-22d3-434a-9575-e60c63fe330a-original-pull-secret\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.682780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.682761    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.682965 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.682949    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:29:18.683023 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.682968    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:29:18.683055 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.683018    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nq6dm\""
Oct 31 07:29:18.683055 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.683040    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ghqlz\""
Oct 31 07:29:18.683176 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.683158    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:29:18.684936 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684918    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7sl8v\""
Oct 31 07:29:18.685011 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684919    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:29:18.685011 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684970    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.685011 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684983    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:29:18.685011 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684994    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:29:18.685206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684988    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.685206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.684985    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.685206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.685053    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:29:18.685206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.685070    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:29:18.685206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.685075    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.685463 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.685446    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:29:18.686696 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686681    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:18.686787 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686744    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zjbzq\""
Oct 31 07:29:18.686787 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686750    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k6p7p\""
Oct 31 07:29:18.686787 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686778    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frjtn\""
Oct 31 07:29:18.686942 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686805    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hfwqd\""
Oct 31 07:29:18.686942 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686894    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:29:18.686942 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686900    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xr95\""
Oct 31 07:29:18.686942 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.686899    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gd7dx\""
Oct 31 07:29:18.688324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.688311    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x8xx7\""
Oct 31 07:29:18.688388 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.688374    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:29:18.688767 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.688753    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcgd\" (UniqueName: \"kubernetes.io/projected/a1503777-02b6-4287-86f7-e1b7094397f2-kube-api-access-gmcgd\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.717131 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717093    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:29:18.717131 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717099    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717158    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717165    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717253    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717286    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717293    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717291    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717308    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717303    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:18.717348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717352    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.717976 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717384    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.717976 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717443    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:29:18.717976 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.717903    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:29:18.733081 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.733051    4371 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:18.733643 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.733622    4371 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:18.748828 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.748798    4371 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:18.748959 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:18.748853    4371 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:18.748959 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.748804    4371 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:18.748959 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:18.748955    4371 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-east-2.compute.internal"
Oct 31 07:29:18.770639 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.770610    4371 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:29:18.781775 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781750    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.781868 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781782    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.781868 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781799    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.781868 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781820    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.781982 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781863    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.781982 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781896    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.781982 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781919    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.782120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.781996    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.782120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782059    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.782215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782121    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.782215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782147    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:18.782215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782200    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.782324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782245    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.782324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782276    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.782324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782314    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.782453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782336    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782378    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782415    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782413    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-config\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.782631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782478    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.782631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782502    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.782631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782564    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782738 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782628    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782738 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782662    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b89ab622-9b1b-4954-924b-b75da49f162f-serviceca\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.782738 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782665    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9g5\" (UniqueName: \"kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782786    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.782909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782829    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.782909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782856    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:29:18.782909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782858    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-daemon-config\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.782909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782882    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstks\" (UniqueName: \"kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.783146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.782942    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.783146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783017    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.783146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783077    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49be424d-4007-47e0-97e2-e8b63e9af6eb-iptables-alerter-script\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.783146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783084    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnhv\" (UniqueName: \"kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.783146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783099    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-tmp-dir\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.783355 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783150    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67890a7c-54d4-4af5-a01c-fee014b737a8-konnectivity-ca\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:29:18.783355 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783165    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.783355 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783238    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:29:18.783355 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783278    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.783548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783411    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpflk\" (UniqueName: \"kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.783548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783450    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.783548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783478    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.783548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783531    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67890a7c-54d4-4af5-a01c-fee014b737a8-agent-certs\") pod \"konnectivity-agent-5dvbr\" (UID: \"67890a7c-54d4-4af5-a01c-fee014b737a8\") " pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:29:18.783548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783541    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.783792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783668    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8g9w\" (UniqueName: \"kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.783792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783667    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.783792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783696    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqtm\" (UniqueName: \"kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.783792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783777    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783800    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-config-volume\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783806    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783853    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783874    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783889    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783907    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783940    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.783956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.783948    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cni-binary-copy\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784011    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-kubelet-config\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784045    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784071    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784103    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784128    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784151    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784176    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784199    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784224    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784248    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784260    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-env-overrides\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.784294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784274    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784325    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-cnibin\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784413    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784485    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784554    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb9197ad-22d3-434a-9575-e60c63fe330a-dbus\") pod \"global-pull-secret-syncer-xgqwg\" (UID: \"bb9197ad-22d3-434a-9575-e60c63fe330a\") " pod="kube-system/global-pull-secret-syncer-xgqwg"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784556    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-tmp\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784570    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt7n\" (UniqueName: \"kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784591    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784636    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784659    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784672    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784689    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1503777-02b6-4287-86f7-e1b7094397f2-system-cni-dir\") pod \"multus-additional-cni-plugins-gxcql\" (UID: \"a1503777-02b6-4287-86f7-e1b7094397f2\") " pod="openshift-multus/multus-additional-cni-plugins-gxcql"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784696    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784726    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4dh\" (UniqueName: \"kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784750    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s4x\" (UniqueName: \"kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784767    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.784780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.784782    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785005    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785026    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785088    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-tuned\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785112    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785135    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785150    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785166    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785192    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785217    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-data-volume\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785237    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785244    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42302c13-04fb-415c-affb-f652ff0be51c-tmp-dir\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785263    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785305    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785322    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785337    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785358    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785385    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xq95\" (UniqueName: \"kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785402    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785553    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785616    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785640    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785640    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovnkube-script-lib\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785656    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvj7\" (UniqueName: \"kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785621    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-cert\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785675    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.785746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785714    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.786130 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.785935    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2484cbbd-3954-4ee8-a0d2-ca0106675dae-ovn-node-metrics-cert\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.788937 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.788916    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnlt\" (UniqueName: \"kubernetes.io/projected/8fbc61ab-9376-4c34-9452-443636430232-kube-api-access-4xnlt\") pod \"network-check-target-56cfh\" (UID: \"8fbc61ab-9376-4c34-9452-443636430232\") " pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:18.789604 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.789585    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstks\" (UniqueName: \"kubernetes.io/projected/41668c10-19d0-4d52-b138-81c2dea94d7f-kube-api-access-nstks\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.789894 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.789878    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9g5\" (UniqueName: \"kubernetes.io/projected/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-kube-api-access-rt9g5\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.791556 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.791486    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqtm\" (UniqueName: \"kubernetes.io/projected/2484cbbd-3954-4ee8-a0d2-ca0106675dae-kube-api-access-8dqtm\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.791556 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.791527    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpflk\" (UniqueName: \"kubernetes.io/projected/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-kube-api-access-tpflk\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.791653 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.791595    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8g9w\" (UniqueName: \"kubernetes.io/projected/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kube-api-access-w8g9w\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.791653 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.791596    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnhv\" (UniqueName: \"kubernetes.io/projected/49be424d-4007-47e0-97e2-e8b63e9af6eb-kube-api-access-bjnhv\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.792050 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.792034    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt7n\" (UniqueName: \"kubernetes.io/projected/6eb4776c-2843-4b41-ab1e-1e65fa89ff36-kube-api-access-hrt7n\") pod \"ingress-canary-x2pmn\" (UID: \"6eb4776c-2843-4b41-ab1e-1e65fa89ff36\") " pod="openshift-ingress-canary/ingress-canary-x2pmn"
Oct 31 07:29:18.795092 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.795073    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4dh\" (UniqueName: \"kubernetes.io/projected/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-kube-api-access-7c4dh\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.795266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.795251    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xq95\" (UniqueName: \"kubernetes.io/projected/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-kube-api-access-5xq95\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.795728 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.795712    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s4x\" (UniqueName: \"kubernetes.io/projected/42302c13-04fb-415c-affb-f652ff0be51c-kube-api-access-z2s4x\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.795797 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.795781    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvj7\" (UniqueName: \"kubernetes.io/projected/b89ab622-9b1b-4954-924b-b75da49f162f-kube-api-access-nvvj7\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.886546 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886492    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886562    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886576    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-log-socket\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886591    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886624    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886646    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886660    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886669    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b89ab622-9b1b-4954-924b-b75da49f162f-host\") pod \"node-ca-kkwsp\" (UID: \"b89ab622-9b1b-4954-924b-b75da49f162f\") " pod="openshift-image-registry/node-ca-kkwsp"
Oct 31 07:29:18.886707 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886678    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:18.886711    4371 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886715    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-etc-kubernetes\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886750    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-kubernetes\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886712    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886760    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886754    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-run\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886762    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-socket-dir-parent\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:18.886800    4371 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:19.386778159 +0000 UTC m=+1.802908832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886845    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886866    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886883    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886903    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886922    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886926    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-sys\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886928    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886964    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886966    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-host\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886987    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.886967    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-registration-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887002    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887011    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887043    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-cnibin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887049    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.887065 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887070    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887093    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887097    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-netns\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887130    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887148    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887180    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887207    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887210    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-lib-modules\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887232    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887254    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-kubelet\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887258    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887255    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-run-multus-certs\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887231    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887275    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-sys-fs\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887287    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887293    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-run-netns\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887262    4371 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887308    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887319    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-device-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887334    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887349    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-systemd-units\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887370    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-ovn\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887378    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-kubelet\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887355    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887411    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887423    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-systemd\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887432    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887449    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887471    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887490    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-hostroot\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887493    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887498    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887550    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887553    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysconfig\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887566    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-modprobe-d\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887583    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887692    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887724    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887750    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887765    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-bin\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887776    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887782    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-etc-sysctl-conf\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887815    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42302c13-04fb-415c-affb-f652ff0be51c-hosts-file\") pod \"node-resolver-xg255\" (UID: \"42302c13-04fb-415c-affb-f652ff0be51c\") " pod="openshift-dns/node-resolver-xg255"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887871    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-crio-socket\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887886    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-multus-conf-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887896    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887910    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-host-var-lib-cni-multus\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887932    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887941    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-bin\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887966    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-cni-netd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887970    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.887997    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888018    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888040    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.888004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888044    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-socket-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888065    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888074    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49be424d-4007-47e0-97e2-e8b63e9af6eb-host-slash\") pod \"iptables-alerter-89wm7\" (UID: \"49be424d-4007-47e0-97e2-e8b63e9af6eb\") " pod="openshift-network-operator/iptables-alerter-89wm7"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888078    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-systemd\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888091    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888094    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-node-log\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888113    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888154    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b12ceb4-d721-4fe8-9df6-644b2023cd74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnfkd\" (UID: \"4b12ceb4-d721-4fe8-9df6-644b2023cd74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnfkd"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888206    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888230    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/073c9d08-d5d6-4a1b-b39a-fc5a2ab21664-var-lib-kubelet\") pod \"tuned-clpj5\" (UID: \"073c9d08-d5d6-4a1b-b39a-fc5a2ab21664\") " pod="openshift-cluster-node-tuning-operator/tuned-clpj5"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888235    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccd6ee2e-4015-471d-9fe6-b9efc31594ae-system-cni-dir\") pod \"multus-gt6sx\" (UID: \"ccd6ee2e-4015-471d-9fe6-b9efc31594ae\") " pod="openshift-multus/multus-gt6sx"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888274    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888301    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888326    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-var-lib-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888328    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888368    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-run-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888372    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-etc-openvswitch\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.888390    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2484cbbd-3954-4ee8-a0d2-ca0106675dae-host-slash\") pod \"ovnkube-node-lb4zp\" (UID: \"2484cbbd-3954-4ee8-a0d2-ca0106675dae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.889608    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668c10-19d0-4d52-b138-81c2dea94d7f-metrics-certs\") pod \"network-metrics-daemon-6glq2\" (UID: \"41668c10-19d0-4d52-b138-81c2dea94d7f\") " pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.890005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.889688    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4-metrics-tls\") pod \"dns-default-llxtn\" (UID: \"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4\") " pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:18.903991 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.903962    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.975250 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.975224    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glq2"
Oct 31 07:29:18.977433 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:18.977416    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:19.012999 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.012967    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:19.152426 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.152393    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:19.305874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.305806    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6glq2"]
Oct 31 07:29:19.308647 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:19.308619    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41668c10_19d0_4d52_b138_81c2dea94d7f.slice/crio-6df111f5cf2cc6ccfe0daa49661f50c2a37315f113754b891247f750f8a3dbbd WatchSource:0}: Error finding container 6df111f5cf2cc6ccfe0daa49661f50c2a37315f113754b891247f750f8a3dbbd: Status 404 returned error can't find the container with id 6df111f5cf2cc6ccfe0daa49661f50c2a37315f113754b891247f750f8a3dbbd
Oct 31 07:29:19.310904 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.310882    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:29:19.329328 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.329271    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-llxtn"]
Oct 31 07:29:19.332807 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:19.332783    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8200a7_46cd_4b23_a1f2_1acdb0dd67d4.slice/crio-9868a544531fe369cfc29518c3122be56e77dc4f34fcebd4a3d9ec65fb770d03 WatchSource:0}: Error finding container 9868a544531fe369cfc29518c3122be56e77dc4f34fcebd4a3d9ec65fb770d03: Status 404 returned error can't find the container with id 9868a544531fe369cfc29518c3122be56e77dc4f34fcebd4a3d9ec65fb770d03
Oct 31 07:29:19.391576 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.391540    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:19.391729 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:19.391651    4371 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:19.391729 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:19.391709    4371 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.391696202 +0000 UTC m=+2.807826852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:19.736144 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.736104    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-llxtn" event={"ID":"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4","Type":"ContainerStarted","Data":"9868a544531fe369cfc29518c3122be56e77dc4f34fcebd4a3d9ec65fb770d03"}
Oct 31 07:29:19.737298 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.737265    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glq2" event={"ID":"41668c10-19d0-4d52-b138-81c2dea94d7f","Type":"ContainerStarted","Data":"6df111f5cf2cc6ccfe0daa49661f50c2a37315f113754b891247f750f8a3dbbd"}
Oct 31 07:29:19.754202 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.754172    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:19.954909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:19.954874    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.355097 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:20.355016    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.405249 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:20.405215    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:20.405432 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:20.405341    4371 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:20.405432 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:20.405406    4371 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls podName:0b2784ec-df9b-4a2e-8dc6-4af7d170341a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:22.405387921 +0000 UTC m=+4.821518590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd96w" (UID: "0b2784ec-df9b-4a2e-8dc6-4af7d170341a") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:20.560341 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:20.560304    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.159661 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.159575    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.743572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.743543    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-llxtn" event={"ID":"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4","Type":"ContainerStarted","Data":"d4d8eb56b18919e748b9d44801c0536f6740f48d6946c5519b12ad6ff67397a2"}
Oct 31 07:29:21.745297 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.745266    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glq2" event={"ID":"41668c10-19d0-4d52-b138-81c2dea94d7f","Type":"ContainerStarted","Data":"c296d753b99b433e93353bf2641ba5297f3cdec84ab5304485e70309bf6d306f"}
Oct 31 07:29:21.745398 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.745322    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glq2" event={"ID":"41668c10-19d0-4d52-b138-81c2dea94d7f","Type":"ContainerStarted","Data":"5610e866c0a5153656cd152f527750c33dfae56b967b8c57d679674ea0532936"}
Oct 31 07:29:21.764272 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.764236    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.998936 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:21.998902    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.366120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.366034    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.419917 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.419886    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:22.422187 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.422159    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b2784ec-df9b-4a2e-8dc6-4af7d170341a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd96w\" (UID: \"0b2784ec-df9b-4a2e-8dc6-4af7d170341a\") " pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:22.572711 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.572678    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.577600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.577580    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd96w"
Oct 31 07:29:22.750482 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.750446    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-llxtn" event={"ID":"2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4","Type":"ContainerStarted","Data":"f60513e26540a4e0022de2be921704df4f33dcc3c41c73f2878916e23a7230ad"}
Oct 31 07:29:22.845687 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.845653    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wd96w"]
Oct 31 07:29:22.849196 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:22.849169    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2784ec_df9b_4a2e_8dc6_4af7d170341a.slice/crio-94429961095b7f615a013c8f4652986fa3327f348bd60a295404a03af3ec6f5a WatchSource:0}: Error finding container 94429961095b7f615a013c8f4652986fa3327f348bd60a295404a03af3ec6f5a: Status 404 returned error can't find the container with id 94429961095b7f615a013c8f4652986fa3327f348bd60a295404a03af3ec6f5a
Oct 31 07:29:22.960095 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:22.960066    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:23.158103 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.158016    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:23.614974 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.614945    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:23.755337 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.755301    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd96w" event={"ID":"0b2784ec-df9b-4a2e-8dc6-4af7d170341a","Type":"ContainerStarted","Data":"f8c62077200a970acfff3c835f3058359ad49252f1d646afc12402384bee04c4"}
Oct 31 07:29:23.755488 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.755358    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd96w" event={"ID":"0b2784ec-df9b-4a2e-8dc6-4af7d170341a","Type":"ContainerStarted","Data":"e85a74ccb1db60b8358c568c0e0599f41929f242ac2b92d99821f132969849bb"}
Oct 31 07:29:23.755488 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.755377    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd96w" event={"ID":"0b2784ec-df9b-4a2e-8dc6-4af7d170341a","Type":"ContainerStarted","Data":"94429961095b7f615a013c8f4652986fa3327f348bd60a295404a03af3ec6f5a"}
Oct 31 07:29:23.755488 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:23.755478    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:24.150053 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.149966    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.258324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.258292    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.362934 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.362897    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.460825 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.460748    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.565183 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.565148    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.832830 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:24.832797    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:25.763922 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:25.763880    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd96w" event={"ID":"0b2784ec-df9b-4a2e-8dc6-4af7d170341a","Type":"ContainerStarted","Data":"6d2c9e3f7d89c00745630ffb4c173cb11c0703a7425c538d53679b14e9f992f6"}
Oct 31 07:29:26.360248 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:26.360219    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:26.360650 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:26.360394    4371 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:26.593127 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:26.593100    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:26.700222 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:26.700163    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.122746 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.122710    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7s9kp"]
Oct 31 07:29:27.124938 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.124919    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.131467 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.131442    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 07:29:27.131592 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.131483    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-pb4z6\""
Oct 31 07:29:27.131634 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.131618    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 07:29:27.137609 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.137585    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7s9kp"]
Oct 31 07:29:27.222108 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.222072    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"]
Oct 31 07:29:27.224172 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.224155    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65ddfff67-pdktj"]
Oct 31 07:29:27.224285 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.224267    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:27.225792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.225770    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.231058 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.231036    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5l7jx\""
Oct 31 07:29:27.231152 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.231131    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6ctpv\""
Oct 31 07:29:27.231415 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.231399    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 07:29:27.235455 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.235434    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 07:29:27.236395 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.236370    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"]
Oct 31 07:29:27.236974 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.236957    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.240787 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.240745    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 07:29:27.241151 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.241129    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 07:29:27.241295 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.241177    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 07:29:27.241740 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.241721    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65ddfff67-pdktj"]
Oct 31 07:29:27.253391 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.253360    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317e3dea-65df-4a6a-917b-088d12b5fef3-tmp\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.253501 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.253398    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jdr\" (UniqueName: \"kubernetes.io/projected/317e3dea-65df-4a6a-917b-088d12b5fef3-kube-api-access-28jdr\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.354280 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354242    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-image-registry-private-configuration\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354280 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354284    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-installation-pull-secrets\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354307    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftd4m\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-kube-api-access-ftd4m\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354367    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317e3dea-65df-4a6a-917b-088d12b5fef3-tmp\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354403    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-registry-tls\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354421    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-registry-certificates\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354463    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jdr\" (UniqueName: \"kubernetes.io/projected/317e3dea-65df-4a6a-917b-088d12b5fef3-kube-api-access-28jdr\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.354561 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354544    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-trusted-ca\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354587    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-bound-sa-token\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354616    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5afcaf-57e8-4957-a9ce-766f71811769-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8z45m\" (UID: \"ca5afcaf-57e8-4957-a9ce-766f71811769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:27.354817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354744    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb292afd-74af-4692-ab45-c834ed320600-ca-trust-extracted\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.354817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.354785    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317e3dea-65df-4a6a-917b-088d12b5fef3-tmp\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.358613 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.358591    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.364119 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.364084    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jdr\" (UniqueName: \"kubernetes.io/projected/317e3dea-65df-4a6a-917b-088d12b5fef3-kube-api-access-28jdr\") pod \"downloads-6dd789c8b7-7s9kp\" (UID: \"317e3dea-65df-4a6a-917b-088d12b5fef3\") " pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.436074 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.435993    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:27.456015 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.455986    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb292afd-74af-4692-ab45-c834ed320600-ca-trust-extracted\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456160 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456027    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-image-registry-private-configuration\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456160 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456046    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-installation-pull-secrets\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456160 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456063    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftd4m\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-kube-api-access-ftd4m\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456303 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456255    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-registry-tls\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456303 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456277    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-registry-certificates\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456303 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456295    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-trusted-ca\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456451 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456320    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-bound-sa-token\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.456451 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456343    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5afcaf-57e8-4957-a9ce-766f71811769-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8z45m\" (UID: \"ca5afcaf-57e8-4957-a9ce-766f71811769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:27.456582 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.456461    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb292afd-74af-4692-ab45-c834ed320600-ca-trust-extracted\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.457798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.457323    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-registry-certificates\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.457798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.457750    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb292afd-74af-4692-ab45-c834ed320600-trusted-ca\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.458804 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.458783    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-image-registry-private-configuration\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.458804 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.458792    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb292afd-74af-4692-ab45-c834ed320600-installation-pull-secrets\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.459122 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.459100    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5afcaf-57e8-4957-a9ce-766f71811769-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8z45m\" (UID: \"ca5afcaf-57e8-4957-a9ce-766f71811769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:27.459226 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.459210    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-registry-tls\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.464439 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.464412    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-bound-sa-token\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.464529 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.464448    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftd4m\" (UniqueName: \"kubernetes.io/projected/bb292afd-74af-4692-ab45-c834ed320600-kube-api-access-ftd4m\") pod \"image-registry-65ddfff67-pdktj\" (UID: \"bb292afd-74af-4692-ab45-c834ed320600\") " pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.536080 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.536038    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:27.543469 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.543447    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:27.706083 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.706061    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7s9kp"]
Oct 31 07:29:27.708641 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:27.708618    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317e3dea_65df_4a6a_917b_088d12b5fef3.slice/crio-be63492c54b1dac6eb5c01545f5037dc586c71f1274dd91f5cb64be73e45b628 WatchSource:0}: Error finding container be63492c54b1dac6eb5c01545f5037dc586c71f1274dd91f5cb64be73e45b628: Status 404 returned error can't find the container with id be63492c54b1dac6eb5c01545f5037dc586c71f1274dd91f5cb64be73e45b628
Oct 31 07:29:27.772101 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.771726    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7s9kp" event={"ID":"317e3dea-65df-4a6a-917b-088d12b5fef3","Type":"ContainerStarted","Data":"be63492c54b1dac6eb5c01545f5037dc586c71f1274dd91f5cb64be73e45b628"}
Oct 31 07:29:27.807047 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.807009    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"]
Oct 31 07:29:27.809924 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:27.809898    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5afcaf_57e8_4957_a9ce_766f71811769.slice/crio-8414571967c91191dcbf08951444db699b036ddcc4eb83220aae90576ee41ed2 WatchSource:0}: Error finding container 8414571967c91191dcbf08951444db699b036ddcc4eb83220aae90576ee41ed2: Status 404 returned error can't find the container with id 8414571967c91191dcbf08951444db699b036ddcc4eb83220aae90576ee41ed2
Oct 31 07:29:27.821826 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:27.821804    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65ddfff67-pdktj"]
Oct 31 07:29:27.824073 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:27.824044    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb292afd_74af_4692_ab45_c834ed320600.slice/crio-23e9564ea0198d620122051f91130b1346075e8b8430739f64a8e1fe6a16c2d3 WatchSource:0}: Error finding container 23e9564ea0198d620122051f91130b1346075e8b8430739f64a8e1fe6a16c2d3: Status 404 returned error can't find the container with id 23e9564ea0198d620122051f91130b1346075e8b8430739f64a8e1fe6a16c2d3
Oct 31 07:29:28.423038 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:28.423000    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:28.776257 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:28.776217    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ddfff67-pdktj" event={"ID":"bb292afd-74af-4692-ab45-c834ed320600","Type":"ContainerStarted","Data":"f767ff7d044fcc6395cd3568501d9b1dfcb3619de761f4dcf7a0123ba2932d23"}
Oct 31 07:29:28.776421 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:28.776284    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ddfff67-pdktj" event={"ID":"bb292afd-74af-4692-ab45-c834ed320600","Type":"ContainerStarted","Data":"23e9564ea0198d620122051f91130b1346075e8b8430739f64a8e1fe6a16c2d3"}
Oct 31 07:29:28.777552 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:28.777527    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m" event={"ID":"ca5afcaf-57e8-4957-a9ce-766f71811769","Type":"ContainerStarted","Data":"8414571967c91191dcbf08951444db699b036ddcc4eb83220aae90576ee41ed2"}
Oct 31 07:29:28.797172 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:28.797113    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65ddfff67-pdktj" podStartSLOduration=1.797095787 podStartE2EDuration="1.797095787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:28.796683043 +0000 UTC m=+11.212813717" watchObservedRunningTime="2026-10-31 07:29:28.797095787 +0000 UTC m=+11.213226460"
Oct 31 07:29:29.783299 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:29.783267    4371 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:29.783765 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:29.783263    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m" event={"ID":"ca5afcaf-57e8-4957-a9ce-766f71811769","Type":"ContainerStarted","Data":"a75cbb5ec4171d48f45b3a5da519ff68e4cee697e788fa23d82d9a45f81c1cb1"}
Oct 31 07:29:29.794670 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:29.794631    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:29:29.799765 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:29.799720    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m" podStartSLOduration=1.828083803 podStartE2EDuration="2.79970557s" totalImagesPullingTime="971.621767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:29.799329166 +0000 UTC m=+12.215459843" watchObservedRunningTime="2026-10-31 07:29:29.79970557 +0000 UTC m=+12.215836242"
Oct 31 07:29:30.786541 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:30.786497    4371 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:32.995386 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:32.995354    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lb4zp"
Oct 31 07:29:33.762071 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:33.762038    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-llxtn"
Oct 31 07:29:33.911149 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:33.911118    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:34.812446 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.812413    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:29:34.816982 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.816961    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.824908 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.824883    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 07:29:34.824908 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.824906    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 07:29:34.825088 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.824975    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 07:29:34.825088 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.825082    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 07:29:34.825191 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.825100    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vtff4\""
Oct 31 07:29:34.825191 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.825169    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 07:29:34.825919 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.825895    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:29:34.911374 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911340    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911389    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911418    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911447    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911474    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911503    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:34.911601 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:34.911548    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f8d\" (UniqueName: \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.012717 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.012681    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.012901 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.012726    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.012901 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.012751    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013087 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013057    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013087 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013112    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013163    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013193    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f8d\" (UniqueName: \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013439    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013461    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013651 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013545    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.013651 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.013592    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.015805 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.015780    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.015892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.015820    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.021940 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.021917    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f8d\" (UniqueName: \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\") pod \"console-5cfb54d5bb-m8d7q\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") " pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.133004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.132910    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:35.418533 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.418437    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:29:35.422610 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:35.422575    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe4e977_e165_4d0a_ac97_8ed88dcaef4e.slice/crio-6cb626f00fc99b2e407495ccf841a07c54973bc1ade9912351620b933fb51c61 WatchSource:0}: Error finding container 6cb626f00fc99b2e407495ccf841a07c54973bc1ade9912351620b933fb51c61: Status 404 returned error can't find the container with id 6cb626f00fc99b2e407495ccf841a07c54973bc1ade9912351620b933fb51c61
Oct 31 07:29:35.808482 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:35.808437    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfb54d5bb-m8d7q" event={"ID":"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e","Type":"ContainerStarted","Data":"6cb626f00fc99b2e407495ccf841a07c54973bc1ade9912351620b933fb51c61"}
Oct 31 07:29:36.056109 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.056071    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:36.070583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.070139    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:36.070583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.070287    4371 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:36.074481 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.074456    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8z45m"
Oct 31 07:29:36.156650 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.156616    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:36.991985 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.991951    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"]
Oct 31 07:29:36.995195 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.995157    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:36.995496 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:36.995475    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.003631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.003607    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 07:29:37.003808 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.003778    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8h2xg\""
Oct 31 07:29:37.005076 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.005053    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 07:29:37.005187 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.005139    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:29:37.005228 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.005216    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 07:29:37.005312 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.005293    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:29:37.005757 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.005690    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"]
Oct 31 07:29:37.102917 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.102884    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:37.134660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.134625    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.134842 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.134671    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32078f71-611e-416e-aab0-02709b28be86-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.134842 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.134696    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlp4p\" (UniqueName: \"kubernetes.io/projected/32078f71-611e-416e-aab0-02709b28be86-kube-api-access-nlp4p\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.134959 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.134830    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.194102 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.194062    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:37.236138 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.236092    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.236138 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.236139    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32078f71-611e-416e-aab0-02709b28be86-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.236382 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.236170    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlp4p\" (UniqueName: \"kubernetes.io/projected/32078f71-611e-416e-aab0-02709b28be86-kube-api-access-nlp4p\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.236382 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.236211    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.237292 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.237268    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32078f71-611e-416e-aab0-02709b28be86-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.239039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.239012    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.239138 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.239044    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32078f71-611e-416e-aab0-02709b28be86-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.246575 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.246500    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlp4p\" (UniqueName: \"kubernetes.io/projected/32078f71-611e-416e-aab0-02709b28be86-kube-api-access-nlp4p\") pod \"prometheus-operator-9cd89b74c-nnmtp\" (UID: \"32078f71-611e-416e-aab0-02709b28be86\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.314167 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.313896    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"
Oct 31 07:29:37.596252 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.596127    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp"]
Oct 31 07:29:37.599774 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:37.599742    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32078f71_611e_416e_aab0_02709b28be86.slice/crio-04fc2275718496b5b27d7e1daf2049ee9b560cc4edf7189c1a81878e990f0658 WatchSource:0}: Error finding container 04fc2275718496b5b27d7e1daf2049ee9b560cc4edf7189c1a81878e990f0658: Status 404 returned error can't find the container with id 04fc2275718496b5b27d7e1daf2049ee9b560cc4edf7189c1a81878e990f0658
Oct 31 07:29:37.817713 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:37.817666    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp" event={"ID":"32078f71-611e-416e-aab0-02709b28be86","Type":"ContainerStarted","Data":"04fc2275718496b5b27d7e1daf2049ee9b560cc4edf7189c1a81878e990f0658"}
Oct 31 07:29:38.823543 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:38.823487    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfb54d5bb-m8d7q" event={"ID":"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e","Type":"ContainerStarted","Data":"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"}
Oct 31 07:29:38.871912 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:38.871852    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cfb54d5bb-m8d7q" podStartSLOduration=1.906678114 podStartE2EDuration="4.871834362s" totalImagesPullingTime="2.965156248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:38.868062558 +0000 UTC m=+21.284193231" watchObservedRunningTime="2026-10-31 07:29:38.871834362 +0000 UTC m=+21.287965036"
Oct 31 07:29:39.314225 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:39.314186    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.830041 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:39.829999    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp" event={"ID":"32078f71-611e-416e-aab0-02709b28be86","Type":"ContainerStarted","Data":"5d0ab50526e483faf1dd184ce3b948b97e7f51c68c372a1e6536f28546c6eac1"}
Oct 31 07:29:39.830479 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:39.830065    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp" event={"ID":"32078f71-611e-416e-aab0-02709b28be86","Type":"ContainerStarted","Data":"c6e186d59cab61dc5735d20c9a2fadf519c592f00e26310524c4b743f91c189c"}
Oct 31 07:29:39.850540 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:39.850467    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nnmtp" podStartSLOduration=2.491855321 podStartE2EDuration="3.850453346s" totalImagesPullingTime="1.358598025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:39.848602488 +0000 UTC m=+22.264733166" watchObservedRunningTime="2026-10-31 07:29:39.850453346 +0000 UTC m=+22.266584016"
Oct 31 07:29:41.140189 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.140148    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.263475 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.263441    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.335279 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.335247    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"]
Oct 31 07:29:41.338913 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.338891    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.351084 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.351058    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"]
Oct 31 07:29:41.351370 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.351286    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fdg5m\""
Oct 31 07:29:41.351370 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.351334    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 07:29:41.351370 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.351356    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 07:29:41.375213 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.375173    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mrgr9"]
Oct 31 07:29:41.378366 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.378330    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.387679 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.387631    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 07:29:41.387679 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.387639    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 07:29:41.387834 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.387708    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gsfjw\""
Oct 31 07:29:41.401951 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.401877    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.459915 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.459885    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 07:29:41.468139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468103    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-root\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468149    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-wtmp\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468184    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-sys\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468208    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-metrics-client-ca\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468288    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rh9\" (UniqueName: \"kubernetes.io/projected/d8a954e8-34d3-46a8-89a9-0ec7e005d620-kube-api-access-f2rh9\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468519 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468324    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.468519 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468388    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-textfile\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468519 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468405    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-tls\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468519 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468422    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwmj\" (UniqueName: \"kubernetes.io/projected/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-kube-api-access-pvwmj\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.468519 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468458    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468778 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468570    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.468778 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468676    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.468778 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.468708    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.514255 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.514210    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.569464 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569420    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-root\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569474    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-wtmp\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569521    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-sys\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569537    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-metrics-client-ca\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569561    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rh9\" (UniqueName: \"kubernetes.io/projected/d8a954e8-34d3-46a8-89a9-0ec7e005d620-kube-api-access-f2rh9\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569560    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-root\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569587    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569614    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-sys\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569633    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-textfile\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.569668 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569661    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-tls\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569689    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwmj\" (UniqueName: \"kubernetes.io/projected/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-kube-api-access-pvwmj\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569715    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-wtmp\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569726    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569754    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569799    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.570018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.569821    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.570305 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.570148    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-textfile\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570305 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.570267    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-metrics-client-ca\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570576 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.570552    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.570701 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.570558    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.572525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.572486    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-tls\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.572783 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.572758    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.572892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.572843    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8a954e8-34d3-46a8-89a9-0ec7e005d620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.572964 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.572940    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.578024 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.578004    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rh9\" (UniqueName: \"kubernetes.io/projected/d8a954e8-34d3-46a8-89a9-0ec7e005d620-kube-api-access-f2rh9\") pod \"node-exporter-mrgr9\" (UID: \"d8a954e8-34d3-46a8-89a9-0ec7e005d620\") " pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.578902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.578878    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwmj\" (UniqueName: \"kubernetes.io/projected/7c2e8f6a-5c3b-4404-b98f-c0e3519d8841-kube-api-access-pvwmj\") pod \"openshift-state-metrics-7bd7489957-6wbfw\" (UID: \"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.623672 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.623617    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.658716 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.658633    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"
Oct 31 07:29:41.697588 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.697548    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrgr9"
Oct 31 07:29:41.744027 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.734952    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.839568 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.839518    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrgr9" event={"ID":"d8a954e8-34d3-46a8-89a9-0ec7e005d620","Type":"ContainerStarted","Data":"039033a2b0beca6f6d664ae7b1432e6d6b53c9140ed3171af9626d9897a69ae7"}
Oct 31 07:29:41.843832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.843795    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.944649 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.944598    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw"]
Oct 31 07:29:41.947676 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:41.947641    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2e8f6a_5c3b_4404_b98f_c0e3519d8841.slice/crio-31770aa07c289ff69ebb26cc3f3df7a849d0551ab47d80883f2582cc2c41bb4c WatchSource:0}: Error finding container 31770aa07c289ff69ebb26cc3f3df7a849d0551ab47d80883f2582cc2c41bb4c: Status 404 returned error can't find the container with id 31770aa07c289ff69ebb26cc3f3df7a849d0551ab47d80883f2582cc2c41bb4c
Oct 31 07:29:41.980485 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:41.980456    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.360541 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.360432    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.566426 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.566349    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.783130 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.783098    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.845202 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.845165    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw" event={"ID":"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841","Type":"ContainerStarted","Data":"c8634b718687f419a55de3d11e862d544914ae85d248028da2fcc07d0b22681c"}
Oct 31 07:29:42.845375 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.845225    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw" event={"ID":"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841","Type":"ContainerStarted","Data":"4bbf2c14a554f8776ab2a7401cdad6ca8842c9562156802970ccda65f901d433"}
Oct 31 07:29:42.845375 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.845244    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw" event={"ID":"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841","Type":"ContainerStarted","Data":"31770aa07c289ff69ebb26cc3f3df7a849d0551ab47d80883f2582cc2c41bb4c"}
Oct 31 07:29:42.885196 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:42.885160    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.455041 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.455012    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d78f7778-sxmhc"]
Oct 31 07:29:43.465574 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.465551    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.469945 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.469663    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d78f7778-sxmhc"]
Oct 31 07:29:43.470752 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.470733    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 07:29:43.471846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.471671    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 07:29:43.471846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.471674    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 07:29:43.471846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.471764    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dsssz\""
Oct 31 07:29:43.471846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.471780    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 07:29:43.471846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.471838    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 07:29:43.473417 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.473394    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7ioi3aik0if88\""
Oct 31 07:29:43.587209 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587172    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587209 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587212    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587462 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587236    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c4a477-81cf-495f-bcfe-ffc53cf532e8-tmp\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587462 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587320    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587463    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587504    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhpn\" (UniqueName: \"kubernetes.io/projected/64c4a477-81cf-495f-bcfe-ffc53cf532e8-kube-api-access-gxhpn\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587650 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587589    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587650 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587636    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c4a477-81cf-495f-bcfe-ffc53cf532e8-metrics-client-ca\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.587719 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.587660    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-grpc-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688110 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688068    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688123    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688151    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhpn\" (UniqueName: \"kubernetes.io/projected/64c4a477-81cf-495f-bcfe-ffc53cf532e8-kube-api-access-gxhpn\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688180    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688220    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c4a477-81cf-495f-bcfe-ffc53cf532e8-metrics-client-ca\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688243    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-grpc-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688559 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688305    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688559 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688331    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.688559 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688346    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c4a477-81cf-495f-bcfe-ffc53cf532e8-tmp\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.689081 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.688736    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c4a477-81cf-495f-bcfe-ffc53cf532e8-tmp\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.690107 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.690056    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c4a477-81cf-495f-bcfe-ffc53cf532e8-metrics-client-ca\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.691332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.691262    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.691482 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.691333    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.691482 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.691369    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.691630 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.691591    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.692003 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.691983    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.692745 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.692722    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64c4a477-81cf-495f-bcfe-ffc53cf532e8-secret-grpc-tls\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.695967 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.695949    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhpn\" (UniqueName: \"kubernetes.io/projected/64c4a477-81cf-495f-bcfe-ffc53cf532e8-kube-api-access-gxhpn\") pod \"thanos-querier-5d78f7778-sxmhc\" (UID: \"64c4a477-81cf-495f-bcfe-ffc53cf532e8\") " pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.850162 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.850128    4371 generic.go:356] "Generic (PLEG): container finished" podID="d8a954e8-34d3-46a8-89a9-0ec7e005d620" containerID="78faf57a4e611ead807814029fa63b04243a60ed4af9d183de88dd6410a0278b" exitCode=0
Oct 31 07:29:43.850341 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.850214    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrgr9" event={"ID":"d8a954e8-34d3-46a8-89a9-0ec7e005d620","Type":"ContainerDied","Data":"78faf57a4e611ead807814029fa63b04243a60ed4af9d183de88dd6410a0278b"}
Oct 31 07:29:43.850803 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.850779    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:43.852610 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.852583    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw" event={"ID":"7c2e8f6a-5c3b-4404-b98f-c0e3519d8841","Type":"ContainerStarted","Data":"4e8cc5906b6ab86a57717f9e01efdb82e6d21d09b70ed3b120adf17724b461a7"}
Oct 31 07:29:43.888402 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.888356    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6wbfw" podStartSLOduration=1.5089252530000001 podStartE2EDuration="2.888342085s" totalImagesPullingTime="1.379416832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:43.88779947 +0000 UTC m=+26.303930144" watchObservedRunningTime="2026-10-31 07:29:43.888342085 +0000 UTC m=+26.304472757"
Oct 31 07:29:43.917387 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:43.917365    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65ddfff67-pdktj"
Oct 31 07:29:44.103739 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.103650    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.143859 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.143800    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d78f7778-sxmhc"]
Oct 31 07:29:44.147837 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:44.147808    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c4a477_81cf_495f_bcfe_ffc53cf532e8.slice/crio-e505b28cc538374d5fc8b0969b57eb831ed1d0fa9b54cf1bf35421b1e2065379 WatchSource:0}: Error finding container e505b28cc538374d5fc8b0969b57eb831ed1d0fa9b54cf1bf35421b1e2065379: Status 404 returned error can't find the container with id e505b28cc538374d5fc8b0969b57eb831ed1d0fa9b54cf1bf35421b1e2065379
Oct 31 07:29:44.227631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.227482    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.336523 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.336475    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.441383 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.441310    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.548969 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.548936    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.849820 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.849700    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.859720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.859678    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"e505b28cc538374d5fc8b0969b57eb831ed1d0fa9b54cf1bf35421b1e2065379"}
Oct 31 07:29:44.862380 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.862342    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7s9kp" event={"ID":"317e3dea-65df-4a6a-917b-088d12b5fef3","Type":"ContainerStarted","Data":"724e464b34c09aeda90c996d9dfd5a8a6b2a7118235d82f22bb45cc6039533a1"}
Oct 31 07:29:44.863414 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.863395    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:44.867332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.867299    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrgr9" event={"ID":"d8a954e8-34d3-46a8-89a9-0ec7e005d620","Type":"ContainerStarted","Data":"f9b237909c032bbd4313a1ce03ddfc766713385c06ea429537f0803d38250501"}
Oct 31 07:29:44.867453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.867350    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrgr9" event={"ID":"d8a954e8-34d3-46a8-89a9-0ec7e005d620","Type":"ContainerStarted","Data":"f6b8bd790e7d863a5891e202494ea878cddc15369f3409fe81ee2fe5a9e2edcd"}
Oct 31 07:29:44.876210 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.876163    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7s9kp"
Oct 31 07:29:44.885992 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.885923    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7s9kp" podStartSLOduration=1.437918028 podStartE2EDuration="17.885904411s" totalImagesPullingTime="16.447986383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:44.88514625 +0000 UTC m=+27.301276927" watchObservedRunningTime="2026-10-31 07:29:44.885904411 +0000 UTC m=+27.302035082"
Oct 31 07:29:44.932814 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.932548    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mrgr9" podStartSLOduration=2.348623035 podStartE2EDuration="3.932527409s" totalImagesPullingTime="1.498284219s" totalInitContainerRuntime="85.620155ms" isStatefulPod=true podCreationTimestamp="2026-10-31 07:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:44.931274088 +0000 UTC m=+27.347404762" watchObservedRunningTime="2026-10-31 07:29:44.932527409 +0000 UTC m=+27.348658081"
Oct 31 07:29:44.969216 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:44.969122    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:45.133264 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.133181    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:45.133264 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.133248    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:45.139723 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.139654    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:45.656992 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.656955    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:45.769025 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.768978    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:45.877987 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:45.877740    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:29:46.135481 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.135441    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"]
Oct 31 07:29:46.158471 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.158432    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"]
Oct 31 07:29:46.158670 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.158606    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.165882 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.165845    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nl498\""
Oct 31 07:29:46.166063 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.165917    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 07:29:46.180537 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.178363    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:46.283631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.283589    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:46.317913 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.317873    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13f79276-4681-47fc-8a67-91dc6ec86c90-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.318103 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.317936    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tfq\" (UniqueName: \"kubernetes.io/projected/13f79276-4681-47fc-8a67-91dc6ec86c90-kube-api-access-f7tfq\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.419500 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.419420    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13f79276-4681-47fc-8a67-91dc6ec86c90-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.419500 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.419481    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tfq\" (UniqueName: \"kubernetes.io/projected/13f79276-4681-47fc-8a67-91dc6ec86c90-kube-api-access-f7tfq\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.422220 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.422191    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13f79276-4681-47fc-8a67-91dc6ec86c90-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.429084 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.429053    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tfq\" (UniqueName: \"kubernetes.io/projected/13f79276-4681-47fc-8a67-91dc6ec86c90-kube-api-access-f7tfq\") pod \"monitoring-plugin-77f4f76594-vqspw\" (UID: \"13f79276-4681-47fc-8a67-91dc6ec86c90\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.500106 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.500067    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:46.795941 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.795905    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"]
Oct 31 07:29:46.829836 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.829805    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5dvbr"
Oct 31 07:29:46.878853 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:46.878816    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw" event={"ID":"13f79276-4681-47fc-8a67-91dc6ec86c90","Type":"ContainerStarted","Data":"ef8971ee3225480a8f84b02aa68faca45d99c0279595e1281c29c4394c0a415c"}
Oct 31 07:29:47.672205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.671589    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:29:47.705486 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.705449    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:47.715352 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.713646    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.718383 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.718354    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 07:29:47.718383 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.718354    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 07:29:47.719711 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.719691    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 07:29:47.719878 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.719695    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 07:29:47.720105 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.719910    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 07:29:47.720344 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720329    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 07:29:47.720539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720498    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 07:29:47.720637 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720444    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 07:29:47.720717 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720537    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 07:29:47.720770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720384    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2bsxr\""
Oct 31 07:29:47.720770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.720673    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 07:29:47.721202 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.721186    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 07:29:47.721390 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.721376    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 07:29:47.721771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.721719    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5aae1ttr1mkbp\""
Oct 31 07:29:47.721771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.721735    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 07:29:47.728545 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.724632    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:29:47.739333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.739081    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 07:29:47.739333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.739130    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 07:29:47.830461 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830398    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.830461 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830471    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830499    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830589    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830654    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830683    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830724    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830751    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830778    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830810    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830852    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830874    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830900    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830929    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.830995    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.831022    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.831038    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj892\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.831072    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.831096    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.831112 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.831124    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.903899 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.903866    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"eb91394fae4968e04c9705f1004ce9e4b5be17471153584c757bf72f71d7080f"}
Oct 31 07:29:47.904091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.903965    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"415fabc7ccbdf45551924d11b6088d823bd8fb6ce6e1dc06457fe629336e96db"}
Oct 31 07:29:47.904091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.904001    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"26789b20ca918b654fb761f933c9b121cf6f32e54923480bb84b3ccbabf98716"}
Oct 31 07:29:47.932539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932439    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.932539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932491    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.932769 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932542    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.932876 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932846    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.932956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932934    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933014 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.932986    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933069 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933014    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933069 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933059    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933174 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933101    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933174 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933159    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933279 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933192    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933279 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933259    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933304    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933337    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933400    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933437    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933463    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj892\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933471    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933529    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933560    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933587    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.933733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.933609    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.934654 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.934158    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.934872 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.934850    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.937231 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.937206    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.938042 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.938018    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.939989 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.939732    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.940465 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.940440    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.940570 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.940553    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.940756 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.940736    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.941120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.941097    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.942459 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.942423    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.949307 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.949280    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj892\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.956377 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.956346    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.957458 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.957435    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.958160 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.957856    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.958235 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.958179    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.960073 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.960047    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.960228 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.960170    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:47.960480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:47.960434    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\") pod \"prometheus-k8s-0\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:48.196630 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:48.196295    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:48.429387 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:48.428550    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-56cfh"
Oct 31 07:29:48.604525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:48.604401    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:29:48.771633 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:48.771591    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeed61c7_0e8c_417e_b89c_97d1367cd456.slice/crio-05050c5aa390b7f2b0db80bba84395dd0d89e2cfb09984aa050fb886481085c1 WatchSource:0}: Error finding container 05050c5aa390b7f2b0db80bba84395dd0d89e2cfb09984aa050fb886481085c1: Status 404 returned error can't find the container with id 05050c5aa390b7f2b0db80bba84395dd0d89e2cfb09984aa050fb886481085c1
Oct 31 07:29:48.909397 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:48.909336    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"05050c5aa390b7f2b0db80bba84395dd0d89e2cfb09984aa050fb886481085c1"}
Oct 31 07:29:49.918584 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.918488    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"2d61e556776084dc3761bc660d6ab6bccfbf1e5d13c0183da731d51a13696ee9"}
Oct 31 07:29:49.919067 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.918687    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"9d282cb5c70fae9f93ce1e5e84adc9f183d855d007ca1ccd9d45e0cfa6cadd5b"}
Oct 31 07:29:49.919067 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.918745    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:49.919067 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.918785    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" event={"ID":"64c4a477-81cf-495f-bcfe-ffc53cf532e8","Type":"ContainerStarted","Data":"f0e86895bcb2a22923d27ce9d6b79bf742ec4accac66443b99a28e29ce8e0ddc"}
Oct 31 07:29:49.920976 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.920950    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw" event={"ID":"13f79276-4681-47fc-8a67-91dc6ec86c90","Type":"ContainerStarted","Data":"ddddaf96f76d639dffcd00628deca159df6d97ff5fccc0eba89b1d9c42e595bb"}
Oct 31 07:29:49.921727 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.921702    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:49.927899 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.927864    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw"
Oct 31 07:29:49.948619 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.948562    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc" podStartSLOduration=2.543222188 podStartE2EDuration="6.948545832s" totalImagesPullingTime="4.405323644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:49.946335945 +0000 UTC m=+32.362466649" watchObservedRunningTime="2026-10-31 07:29:49.948545832 +0000 UTC m=+32.364676504"
Oct 31 07:29:49.968003 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:49.967922    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vqspw" podStartSLOduration=1.541564391 podStartE2EDuration="3.967899407s" totalImagesPullingTime="2.426335016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:49.966742049 +0000 UTC m=+32.382872734" watchObservedRunningTime="2026-10-31 07:29:49.967899407 +0000 UTC m=+32.384030075"
Oct 31 07:29:50.927580 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:50.927541    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22" exitCode=0
Oct 31 07:29:50.928175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:50.927633    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"}
Oct 31 07:29:51.864592 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.864552    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:29:51.884444 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.883865    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:29:51.884444 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.883983    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.894359 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.894333    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 07:29:51.979180 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979141    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979208    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979243    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979276    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979314    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979384    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979429    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:51.979689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:51.979456    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhgc\" (UniqueName: \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.080926 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.080889    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.080943    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.080977    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.081008    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.081104    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.081147    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081198 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.081163    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhgc\" (UniqueName: \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.081553 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.081197    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.082056 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.082022    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.082310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.082290    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.083164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.083138    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.083769 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.083745    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.085971 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.085935    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.087013 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.086983    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.100539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.087928    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.100539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.094103    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhgc\" (UniqueName: \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\") pod \"console-8445dbb675-bcnzf\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") " pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.222818 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.222547    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:29:52.538672 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.538495    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:29:52.546950 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:29:52.546835    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c9a4b6_7503_4854_8620_326104eba113.slice/crio-e90bd14cd07a43e17873c9d749b97cb371859faa2c7191bf816e37210717ec9f WatchSource:0}: Error finding container e90bd14cd07a43e17873c9d749b97cb371859faa2c7191bf816e37210717ec9f: Status 404 returned error can't find the container with id e90bd14cd07a43e17873c9d749b97cb371859faa2c7191bf816e37210717ec9f
Oct 31 07:29:52.944631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.943714    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8445dbb675-bcnzf" event={"ID":"13c9a4b6-7503-4854-8620-326104eba113","Type":"ContainerStarted","Data":"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"}
Oct 31 07:29:52.944631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:52.943780    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8445dbb675-bcnzf" event={"ID":"13c9a4b6-7503-4854-8620-326104eba113","Type":"ContainerStarted","Data":"e90bd14cd07a43e17873c9d749b97cb371859faa2c7191bf816e37210717ec9f"}
Oct 31 07:29:54.851744 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:54.851715    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:54.952102 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:54.952061    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:54.956866 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:54.956845    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"}
Oct 31 07:29:55.648693 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.648611    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:55.764409 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.764377    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:55.933814 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.933733    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d78f7778-sxmhc"
Oct 31 07:29:55.964828 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964793    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"}
Oct 31 07:29:55.965039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964870    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"}
Oct 31 07:29:55.965039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964897    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"}
Oct 31 07:29:55.965039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964921    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"}
Oct 31 07:29:55.965039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964943    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerStarted","Data":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"}
Oct 31 07:29:55.965039 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:55.964877    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8445dbb675-bcnzf" podStartSLOduration=4.964863474 podStartE2EDuration="4.964863474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:52.969846103 +0000 UTC m=+35.385976776" watchObservedRunningTime="2026-10-31 07:29:55.964863474 +0000 UTC m=+38.380994143"
Oct 31 07:29:56.002502 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:56.002435    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.41873895 podStartE2EDuration="9.002414283s" totalImagesPullingTime="5.548276801s" totalInitContainerRuntime="35.398532ms" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:55.998527383 +0000 UTC m=+38.414658045" watchObservedRunningTime="2026-10-31 07:29:56.002414283 +0000 UTC m=+38.418544955"
Oct 31 07:29:57.261706 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:57.261676    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:57.366480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:57.366444    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:58.198593 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:58.198552    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:58.866523 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:58.866488    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:58.967772 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:58.967729    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:59.789382 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:29:59.789344    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:29:59.957779 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:29:59.957752    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:00.057485 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:00.057386    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:00.162495 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:00.162460    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:00.262818 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:00.262789    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:01.207809 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:01.207778    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:01.408492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:01.408462    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:02.223361 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:02.223276    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:30:02.223770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:02.223399    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:30:02.227973 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:02.227952    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:30:02.996080 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:02.996048    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:30:03.041739 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:03.041713    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:30:10.287591 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:10.287553    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:10.333307 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:10.333280    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:11.036825 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:11.036798    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:11.578431 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:11.578399    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:11.675451 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:11.675382    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:22.758474 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:22.758427    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:22.865257 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:22.865227    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:22.966288 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:22.966260    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:23.426318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:23.426282    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:23.522901 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:23.522877    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:26.600001 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:26.599966    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:26.688903 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:26.688874    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:26.895938 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:26.895867    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:27.021464 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:27.021432    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:28.061158 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.061094    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5cfb54d5bb-m8d7q" podUID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" containerName="console" containerID="cri-o://37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c" gracePeriod=15
Oct 31 07:30:28.332262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.332204    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:30:28.368064 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368038    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368074    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368093    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368123    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368156    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368173    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368455 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368212    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\" (UniqueName: \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\") pod \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\" (UID: \"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e\") "
Oct 31 07:30:28.368555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368455    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:30:28.368714 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368688    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:28.368770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368721    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:28.368920 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.368900    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:28.370317 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.370286    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:28.370421 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.370381    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "kube-api-access-w2f8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:28.370616 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.370596    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config" pod "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" (UID: "bfe4e977-e165-4d0a-ac97-8ed88dcaef4e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:28.469342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469318    4371 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469340    4371 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-tmp\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469351    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f8d\" (UniqueName: \"kubernetes.io/projected/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-kube-api-access-w2f8d\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469363    4371 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-oauth-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469372    4371 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-service-ca\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469380    4371 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-oauth-serving-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.469532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.469388    4371 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e-console-serving-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:28.530116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:28.530086    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:29.089182 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.089152    4371 generic.go:356] "Generic (PLEG): container finished" podID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" containerID="37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c" exitCode=0
Oct 31 07:30:29.089612 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.089196    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfb54d5bb-m8d7q"
Oct 31 07:30:29.089612 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.089209    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfb54d5bb-m8d7q" event={"ID":"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e","Type":"ContainerDied","Data":"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"}
Oct 31 07:30:29.089612 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.089247    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfb54d5bb-m8d7q" event={"ID":"bfe4e977-e165-4d0a-ac97-8ed88dcaef4e","Type":"ContainerDied","Data":"6cb626f00fc99b2e407495ccf841a07c54973bc1ade9912351620b933fb51c61"}
Oct 31 07:30:29.089612 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.089267    4371 scope.go:122] "RemoveContainer" containerID="37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"
Oct 31 07:30:29.097902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.097886    4371 scope.go:122] "RemoveContainer" containerID="37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"
Oct 31 07:30:29.098207 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:29.098187    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c\": container with ID starting with 37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c not found: ID does not exist" containerID="37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"
Oct 31 07:30:29.098295 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.098213    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c"} err="failed to get container status \"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c\": rpc error: code = NotFound desc = could not find container \"37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c\": container with ID starting with 37d590773dbae41d17eff0386b0cb44fb8560f5c340ef5722f68c713c71bb44c not found: ID does not exist"
Oct 31 07:30:29.115399 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.111267    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:30:29.115399 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.115147    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cfb54d5bb-m8d7q"]
Oct 31 07:30:29.130715 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.130671    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:29.227054 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.227028    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:29.704144 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:29.704102    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" path="/var/lib/kubelet/pods/bfe4e977-e165-4d0a-ac97-8ed88dcaef4e/volumes"
Oct 31 07:30:29.788591 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:29.788556    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:30:35.302929 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.302895    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:35.303557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303199    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="prometheus" containerID="cri-o://d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4" gracePeriod=600
Oct 31 07:30:35.303557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303237    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy" containerID="cri-o://4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c" gracePeriod=600
Oct 31 07:30:35.303557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303258    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="thanos-sidecar" containerID="cri-o://f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76" gracePeriod=600
Oct 31 07:30:35.303557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303291    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-web" containerID="cri-o://c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d" gracePeriod=600
Oct 31 07:30:35.303557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303299    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="config-reloader" containerID="cri-o://affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874" gracePeriod=600
Oct 31 07:30:35.303832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.303306    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b" gracePeriod=600
Oct 31 07:30:35.414789 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.414753    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:35.518653 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.518618    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:35.552986 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.552927    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:35.630727 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630696    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630727 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630731    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630752    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630769    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630803    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630859    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.630907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630894    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630925    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630963    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.630991    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631041    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631067    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631100    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631137    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631177    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631225    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631255    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631318 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631291    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631340    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631374    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"feed61c7-0e8c-417e-b89c-97d1367cd456\" (UID: \"feed61c7-0e8c-417e-b89c-97d1367cd456\") "
Oct 31 07:30:35.631660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631593    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.631822 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.631681    4371 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.632606 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.632324    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:30:35.633048 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.633019    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.633414 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.633388    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.633863 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.633835    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.634243 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.634215    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.634623 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.634590    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.634793 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.634770    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.635099 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.635079    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.635620 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.635592    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.635715 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.635662    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.635873 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.635853    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.636041 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.636018    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.636362 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.636338    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "kube-api-access-jj892". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:35.636486 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.636458    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.636594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.636574    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:30:35.637501 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.637473    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:35.637887 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.637863    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:35.637960 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.637918    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.645969 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.645946    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config" pod "feed61c7-0e8c-417e-b89c-97d1367cd456" (UID: "feed61c7-0e8c-417e-b89c-97d1367cd456"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:35.732262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732236    4371 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-db\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732258    4371 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732269    4371 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732278    4371 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732286    4371 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-tls-assets\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732296    4371 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732305    4371 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732314    4371 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-configmap-metrics-client-ca\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732323    4371 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/feed61c7-0e8c-417e-b89c-97d1367cd456-config-out\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732331    4371 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732339    4371 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/feed61c7-0e8c-417e-b89c-97d1367cd456-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732349    4371 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732358    4371 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-web-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732366    4371 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-grpc-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732375    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj892\" (UniqueName: \"kubernetes.io/projected/feed61c7-0e8c-417e-b89c-97d1367cd456-kube-api-access-jj892\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732383    4371 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732391    4371 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-kube-rbac-proxy\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732400    4371 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:35.732412 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:35.732408    4371 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/feed61c7-0e8c-417e-b89c-97d1367cd456-secret-metrics-client-certs\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:36.116799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116765    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b" exitCode=0
Oct 31 07:30:36.116799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116789    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c" exitCode=0
Oct 31 07:30:36.116799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116797    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d" exitCode=0
Oct 31 07:30:36.116799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116803    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76" exitCode=0
Oct 31 07:30:36.116799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116810    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874" exitCode=0
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116817    4371 generic.go:356] "Generic (PLEG): container finished" podID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4" exitCode=0
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116838    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116857    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116904    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116932    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116956    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116969    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.116979    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.117003    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"}
Oct 31 07:30:36.117192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.117025    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"feed61c7-0e8c-417e-b89c-97d1367cd456","Type":"ContainerDied","Data":"05050c5aa390b7f2b0db80bba84395dd0d89e2cfb09984aa050fb886481085c1"}
Oct 31 07:30:36.124911 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.124896    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.131331 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.131317    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.140599 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.140577    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.141731 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.141501    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:36.144005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.143981    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:36.147955 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.147939    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.154101 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.154074    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.161044 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.161029    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.166275 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166256    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:36.166537 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166522    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="prometheus"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166538    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="prometheus"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166551    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="init-config-reloader"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166556    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="init-config-reloader"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166565    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="config-reloader"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166570    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="config-reloader"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166579    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-web"
Oct 31 07:30:36.166583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166584    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-web"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166594    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="thanos-sidecar"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166599    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="thanos-sidecar"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166605    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-thanos"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166610    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-thanos"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166619    4371 cpu_manager.go:414] "containerMap: removing container" podUID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" containerName="console"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166625    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" containerName="console"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166630    4371 cpu_manager.go:414] "containerMap: removing container" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166635    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166678    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="prometheus"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166688    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="config-reloader"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166694    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-web"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166700    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="thanos-sidecar"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166725    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy-thanos"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166733    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe4e977-e165-4d0a-ac97-8ed88dcaef4e" containerName="console"
Oct 31 07:30:36.166848 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.166738    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" containerName="kube-rbac-proxy"
Oct 31 07:30:36.167370 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.167353    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.167732 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.167716    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.167777 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.167743    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.167777 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.167764    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.168002 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.167986    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.168040 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168007    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.168040 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168020    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.168252 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.168226    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.168307 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168260    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.168307 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168279    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.168641 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.168620    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.168706 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168644    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.168706 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168658    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.168926 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.168904    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.169007 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168934    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.169007 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.168955    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.169239 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.169223    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.169282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.169244    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.169282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.169257    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.169466 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:36.169453    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.169538 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.169468    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.169538 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.169479    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.170154 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170136    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.170224 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170155    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.170292 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170276    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.170379 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170361    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.170428 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170379    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.170796 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170688    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.170796 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170713    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.170975 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170957    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.171036 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.170977    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.171204 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171184    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.171269 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171205    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.171426 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171405    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.171426 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171425    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.171794 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171769    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.171873 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.171797    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.172069 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172050    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.172170 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172070    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.172332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172314    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.172332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172332    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.172584 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172568    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.172584 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172584    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.172832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172815    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.172874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.172831    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.173057 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173042    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.173057 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173056    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.173245 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173226    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.173305 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173245    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.173446 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173428    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.173489 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173446    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.173729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173709    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.173809 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173730    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.173948 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173931    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.173993 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.173948    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.174171 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174153    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.174239 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174173    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.174372 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174353    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.174418 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174373    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.174654 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174636    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.174720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174655    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.174905 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174882    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.174905 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.174904    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.175139 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175122    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.175192 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175140    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.175373 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175358    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.175413 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175373    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.175600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175584    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.175651 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175600    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.175813 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175795    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.175813 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.175812    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.176025 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176007    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.176064 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176026    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.176258 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176235    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.176322 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176260    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.176583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176559    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.176647 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176586    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.176792 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176775    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.176833 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176792    4371 scope.go:122] "RemoveContainer" containerID="4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"
Oct 31 07:30:36.176992 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176972    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b"} err="failed to get container status \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": rpc error: code = NotFound desc = could not find container \"4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b\": container with ID starting with 4aa995a5d9efbd06b33250d3efedc454af65e9a5b85ec14a932da8cba2c4607b not found: ID does not exist"
Oct 31 07:30:36.177032 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.176993    4371 scope.go:122] "RemoveContainer" containerID="4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"
Oct 31 07:30:36.177177 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177157    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c"} err="failed to get container status \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": rpc error: code = NotFound desc = could not find container \"4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c\": container with ID starting with 4ba5e4d8e182ff14becb0b4412cdc100dcde0f07e831416b3fadfcb44fc3f45c not found: ID does not exist"
Oct 31 07:30:36.177177 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177180    4371 scope.go:122] "RemoveContainer" containerID="c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"
Oct 31 07:30:36.177546 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177386    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d"} err="failed to get container status \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": rpc error: code = NotFound desc = could not find container \"c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d\": container with ID starting with c5c90992c691cf01ae1157d4b0b056883de8d453de9c4035390126f11556525d not found: ID does not exist"
Oct 31 07:30:36.177546 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177404    4371 scope.go:122] "RemoveContainer" containerID="f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"
Oct 31 07:30:36.177752 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177729    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76"} err="failed to get container status \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": rpc error: code = NotFound desc = could not find container \"f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76\": container with ID starting with f3b7ff9ad87c55d55f8fda48bcbc8560a603b3188f4862257ef03eb44d41ce76 not found: ID does not exist"
Oct 31 07:30:36.177817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.177752    4371 scope.go:122] "RemoveContainer" containerID="affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"
Oct 31 07:30:36.178207 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178186    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874"} err="failed to get container status \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": rpc error: code = NotFound desc = could not find container \"affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874\": container with ID starting with affb1f4fa4a3c16031fdda17c57058a54aada164a87850f027508bfb1a10f874 not found: ID does not exist"
Oct 31 07:30:36.178207 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178205    4371 scope.go:122] "RemoveContainer" containerID="d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"
Oct 31 07:30:36.178411 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178395    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4"} err="failed to get container status \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": rpc error: code = NotFound desc = could not find container \"d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4\": container with ID starting with d6fb8da35e2221264f6835d3a233c6202c52a71b52f7683d6bfc4fcf2c4498a4 not found: ID does not exist"
Oct 31 07:30:36.178411 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178408    4371 scope.go:122] "RemoveContainer" containerID="651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"
Oct 31 07:30:36.178629 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178612    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 07:30:36.178733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178691    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22"} err="failed to get container status \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": rpc error: code = NotFound desc = could not find container \"651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22\": container with ID starting with 651acbd2e703d9cde3eb4c769fbcaf7a674a2a8bde4a0ee2fadd7c2a2948aa22 not found: ID does not exist"
Oct 31 07:30:36.178800 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178729    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 07:30:36.178985 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.178970    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 07:30:36.179560 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179542    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 07:30:36.179659 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179606    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 07:30:36.179723 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179695    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 07:30:36.179723 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179706    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 07:30:36.179815 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179735    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 07:30:36.179815 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179737    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 07:30:36.179815 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179751    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 07:30:36.179815 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179749    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 07:30:36.179815 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.179773    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 07:30:36.180798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.180779    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2bsxr\""
Oct 31 07:30:36.180879 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.180818    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5aae1ttr1mkbp\""
Oct 31 07:30:36.180879 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.180864    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 07:30:36.190106 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.190070    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 07:30:36.190249 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.190232    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 07:30:36.192337 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.191849    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:36.236744 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236718    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.236865 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236750    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.236865 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236774    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.236865 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236792    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j42w\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-kube-api-access-7j42w\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237026 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236935    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237026 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.236977    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237088 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237033    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237088 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237068    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237092    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237109    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237132    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237265 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237165    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237265 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237228    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237265 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237243    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237265 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237259    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237276    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237347    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237379    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237478 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237451    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.237478 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.237469    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338186 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338159    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338188    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338208    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338226    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338273    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338298    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338332    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338555 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338373    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338763    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338816    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338845    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338875    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.338910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338903    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j42w\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-kube-api-access-7j42w\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338939    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338965    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.338994    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339025    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339060    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339086    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339111    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339134    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339147    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.339670 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.339278    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.340929 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.340279    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.340929 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.340809    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.341874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.341640    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.341874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.341826    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.341874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.341825    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.342343 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.342316    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.342607 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.342527    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.342607 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.342529    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecd0177-02e1-497c-9a18-abb68932318b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.342904 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.342861    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ecd0177-02e1-497c-9a18-abb68932318b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.343108 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.343086    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.343201 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.343152    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.343306 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.343290    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.344055 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.344035    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.344455 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.344429    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-config\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.344664 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.344650    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.344979 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.344956    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ecd0177-02e1-497c-9a18-abb68932318b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.347658 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.347635    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j42w\" (UniqueName: \"kubernetes.io/projected/1ecd0177-02e1-497c-9a18-abb68932318b-kube-api-access-7j42w\") pod \"prometheus-k8s-0\" (UID: \"1ecd0177-02e1-497c-9a18-abb68932318b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.496146 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.496117    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:36.774728 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:36.774698    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:30:36.775559 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:30:36.775533    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecd0177_02e1_497c_9a18_abb68932318b.slice/crio-8e6fbc66c08cee96f961d712b5ce094af387bb1055597943decf7f57d2f9a37c WatchSource:0}: Error finding container 8e6fbc66c08cee96f961d712b5ce094af387bb1055597943decf7f57d2f9a37c: Status 404 returned error can't find the container with id 8e6fbc66c08cee96f961d712b5ce094af387bb1055597943decf7f57d2f9a37c
Oct 31 07:30:37.121682 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:37.121601    4371 generic.go:356] "Generic (PLEG): container finished" podID="1ecd0177-02e1-497c-9a18-abb68932318b" containerID="a1aae4eea870a3b7c96aa22beacca1b2ed594e768b40830f4cd63832dd4a0d94" exitCode=0
Oct 31 07:30:37.121840 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:37.121689    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerDied","Data":"a1aae4eea870a3b7c96aa22beacca1b2ed594e768b40830f4cd63832dd4a0d94"}
Oct 31 07:30:37.121840 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:37.121770    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"8e6fbc66c08cee96f961d712b5ce094af387bb1055597943decf7f57d2f9a37c"}
Oct 31 07:30:37.704392 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:37.704306    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feed61c7-0e8c-417e-b89c-97d1367cd456" path="/var/lib/kubelet/pods/feed61c7-0e8c-417e-b89c-97d1367cd456/volumes"
Oct 31 07:30:38.130437 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130404    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"ce2f064166136d664a73ce813a29b718afb4414bab346c1f0a678b7be9a970bd"}
Oct 31 07:30:38.130625 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130491    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"ce948456418c1a63e9ec72f0ee450d8c6d1005ff11864c13d300e989cdd40fe1"}
Oct 31 07:30:38.130625 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130539    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"8ad7b0d61db416d9efe7e30969b39735b8a21b20bd8f8f047ef9ccbcb42f41d2"}
Oct 31 07:30:38.130625 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130569    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"d2fc97ae2d7ce90b08edaba627b066c92af3dbeaca22e66644d2380aa2438a99"}
Oct 31 07:30:38.130625 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130592    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"51fb8d9ad00219f02b65c58cd28eedc852b522764b7950bf084a5830f7127d67"}
Oct 31 07:30:38.130625 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.130613    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ecd0177-02e1-497c-9a18-abb68932318b","Type":"ContainerStarted","Data":"91583b32961b0c53bc49175487582a7f035b1768af2590e92796a22eac128a86"}
Oct 31 07:30:38.161060 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:38.160784    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.145664288 podStartE2EDuration="2.160768531s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.104243ms" isStatefulPod=false podCreationTimestamp="2026-10-31 07:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:30:38.158784991 +0000 UTC m=+80.574915702" watchObservedRunningTime="2026-10-31 07:30:38.160768531 +0000 UTC m=+80.576899203"
Oct 31 07:30:39.611411 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:39.611380    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:30:41.497201 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:41.497163    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:30:41.600621 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:41.600592    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:41.703501 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:41.703460    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:41.803389 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:30:41.803319    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:59.788923 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:30:59.788887    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:31:01.233700 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:01.233669    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:04.630572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.630492    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8445dbb675-bcnzf" podUID="13c9a4b6-7503-4854-8620-326104eba113" containerName="console" containerID="cri-o://a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8" gracePeriod=15
Oct 31 07:31:04.880727 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.880667    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:31:04.969206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969179    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969213    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969233    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969276    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969465 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969377    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969465 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969457    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969588 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969500    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\" (UniqueName: \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969588 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969550    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\") pod \"13c9a4b6-7503-4854-8620-326104eba113\" (UID: \"13c9a4b6-7503-4854-8620-326104eba113\") "
Oct 31 07:31:04.969753 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969729    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:31:04.969811 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969775    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:31:04.969861 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969831    4371 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-trusted-ca-bundle\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:04.969861 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969826    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:31:04.969941 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969879    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:31:04.969941 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.969885    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:31:04.971387 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.971358    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:31:04.971487 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.971400    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "kube-api-access-zvhgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:31:04.971651 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:04.971485    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert" pod "13c9a4b6-7503-4854-8620-326104eba113" (UID: "13c9a4b6-7503-4854-8620-326104eba113"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:31:05.070988 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.070946    4371 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-oauth-serving-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.070988 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.070983    4371 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-oauth-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.070988 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.070994    4371 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c9a4b6-7503-4854-8620-326104eba113-tmp\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.070988 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.071004    4371 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-console-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.071231 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.071013    4371 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13c9a4b6-7503-4854-8620-326104eba113-service-ca\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.071231 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.071022    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhgc\" (UniqueName: \"kubernetes.io/projected/13c9a4b6-7503-4854-8620-326104eba113-kube-api-access-zvhgc\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.071231 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.071031    4371 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13c9a4b6-7503-4854-8620-326104eba113-console-serving-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:31:05.240951 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.240914    4371 generic.go:356] "Generic (PLEG): container finished" podID="13c9a4b6-7503-4854-8620-326104eba113" containerID="a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8" exitCode=0
Oct 31 07:31:05.241116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.240953    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8445dbb675-bcnzf" event={"ID":"13c9a4b6-7503-4854-8620-326104eba113","Type":"ContainerDied","Data":"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"}
Oct 31 07:31:05.241116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.240962    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8445dbb675-bcnzf"
Oct 31 07:31:05.241116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.241003    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8445dbb675-bcnzf" event={"ID":"13c9a4b6-7503-4854-8620-326104eba113","Type":"ContainerDied","Data":"e90bd14cd07a43e17873c9d749b97cb371859faa2c7191bf816e37210717ec9f"}
Oct 31 07:31:05.241116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.241027    4371 scope.go:122] "RemoveContainer" containerID="a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"
Oct 31 07:31:05.249500 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.249475    4371 scope.go:122] "RemoveContainer" containerID="a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"
Oct 31 07:31:05.249773 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:31:05.249752    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8\": container with ID starting with a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8 not found: ID does not exist" containerID="a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"
Oct 31 07:31:05.249846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.249787    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8"} err="failed to get container status \"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8\": rpc error: code = NotFound desc = could not find container \"a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8\": container with ID starting with a9b6eb63ea48c08016e822578fe0ab7dfd4852c96261a0b00c3a5ef9a7f061a8 not found: ID does not exist"
Oct 31 07:31:05.262547 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.262523    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:31:05.266076 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.266057    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8445dbb675-bcnzf"]
Oct 31 07:31:05.703194 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:05.703120    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c9a4b6-7503-4854-8620-326104eba113" path="/var/lib/kubelet/pods/13c9a4b6-7503-4854-8620-326104eba113/volumes"
Oct 31 07:31:09.846909 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:09.846872    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:29.790005 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:31:29.789969    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:31:36.497178 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:36.497092    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:31:36.512888 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:36.512857    4371 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:31:37.371220 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:37.371195    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:31:59.789629 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:31:59.789592    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:31:59.895223 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:59.895179    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:59.997477 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:31:59.997450    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:01.282034 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:01.281994    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:04.298758 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:04.298721    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:04.387734 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:04.387694    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:29.788951 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:32:29.788911    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:32:37.559552 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:37.559497    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:37.661692 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:37.661661    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:46.779303 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:46.779270    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:46.883883 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:46.883846    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:46.982726 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:32:46.982695    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:59.789527 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:32:59.789472    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:33:01.329157 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:01.329118    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:08.793979 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:08.793940    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:08.913689 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:08.913653    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:14.833040 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:14.833004    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:15.112948 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:15.112867    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:15.209316 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:15.209284    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:15.330251 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:15.330221    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:15.429744 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:15.429666    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:15.968253 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:15.968216    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:16.073739 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:16.073702    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:16.284426 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:16.284337    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:16.480356 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:16.480324    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:16.684332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:16.684238    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:29.790166 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:33:29.790120    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:33:46.804717 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:46.804675    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:47.837123 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:47.837089    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:49.757320 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:49.757288    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:49.859401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:33:49.859368    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:59.790572 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:33:59.790488    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:34:01.370885 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:01.370844    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:08.996623 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:08.996588    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:09.097144 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:09.097109    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:18.273688 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:18.273653    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:29.789064 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:34:29.789025    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:34:37.311207 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:37.311119    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:37.444969 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:37.444941    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:37.546763 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:37.546724    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:37.648823 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:37.648742    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:37.749951 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:37.749924    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:38.548435 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:38.548386    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:38.659012 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:38.658982    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:38.769223 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:34:38.769191    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:59.788662 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:34:59.788619    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:35:01.413967 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:01.413931    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:10.075259 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:10.075221    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:10.167325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:10.167298    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:29.788628 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:35:29.788585    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:35:56.790226 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:56.790195    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:56.931234 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:56.931204    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:57.033574 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:35:57.033543    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:59.789089 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:35:59.789051    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:36:00.370287 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:00.370254    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:00.476548 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:00.476504    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:01.457295 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:01.457261    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:12.992858 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:12.992825    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pt9kz"]
Oct 31 07:36:12.993251 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:12.993142    4371 cpu_manager.go:414] "containerMap: removing container" podUID="13c9a4b6-7503-4854-8620-326104eba113" containerName="console"
Oct 31 07:36:12.993251 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:12.993152    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c9a4b6-7503-4854-8620-326104eba113" containerName="console"
Oct 31 07:36:12.993251 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:12.993207    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c9a4b6-7503-4854-8620-326104eba113" containerName="console"
Oct 31 07:36:12.996182 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:12.996167    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.003724 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.003703    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:36:13.003892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.003766    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 31 07:36:13.003892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.003792    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 07:36:13.004016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.003877    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4r7k\""
Oct 31 07:36:13.004639 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.004623    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:36:13.004874 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.004860    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pt9kz"]
Oct 31 07:36:13.047640 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.047617    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.047752 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.047661    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p69f\" (UniqueName: \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.147987 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.147958    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.148117 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.148008    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p69f\" (UniqueName: \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.148322 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.148304    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.156825 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.156793    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p69f\" (UniqueName: \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\") pod \"s3-init-pt9kz\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") " pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.316151 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.316077    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9kz"
Oct 31 07:36:13.585912 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.585886    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pt9kz"]
Oct 31 07:36:13.588056 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:36:13.588025    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c2df43b_c794_4d6e_aaee_310981189dbd.slice/crio-45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608 WatchSource:0}: Error finding container 45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608: Status 404 returned error can't find the container with id 45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608
Oct 31 07:36:13.589823 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:13.589806    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:36:14.291770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:14.291737    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerStarted","Data":"45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608"}
Oct 31 07:36:17.306793 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:17.306766    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerStarted","Data":"4c15b88cdc5ebabff2c8e9e399ba0ddae1b3e31c5d287e2b93481145332721d5"}
Oct 31 07:36:19.315637 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:19.315598    4371 generic.go:356] "Generic (PLEG): container finished" podID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerID="4c15b88cdc5ebabff2c8e9e399ba0ddae1b3e31c5d287e2b93481145332721d5" exitCode=0
Oct 31 07:36:19.316098 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:19.315683    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerDied","Data":"4c15b88cdc5ebabff2c8e9e399ba0ddae1b3e31c5d287e2b93481145332721d5"}
Oct 31 07:36:23.616378 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:23.616357    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 31 07:36:24.343524 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:24.343469    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerStarted","Data":"b7c5e30e4c33a2765da7b94ab5b3d1810e5748919936a193116b2ec391e97239"}
Oct 31 07:36:24.361174 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:36:24.361128    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-pt9kz" podStartSLOduration=2.906717983 podStartE2EDuration="12.361116323s" totalImagesPullingTime="7.940717772s" totalInitContainerRuntime="1.513680568s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:36:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:36:24.359812658 +0000 UTC m=+426.775943332" watchObservedRunningTime="2026-10-31 07:36:24.361116323 +0000 UTC m=+426.777246995"
Oct 31 07:36:29.789303 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:36:29.789267    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:36:59.789259 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:36:59.789226    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:37:01.498120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:01.498054    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:29.564959 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:29.564922    4371 generic.go:356] "Generic (PLEG): container finished" podID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerID="b7c5e30e4c33a2765da7b94ab5b3d1810e5748919936a193116b2ec391e97239" exitCode=0
Oct 31 07:37:29.565360 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:29.564988    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerDied","Data":"b7c5e30e4c33a2765da7b94ab5b3d1810e5748919936a193116b2ec391e97239"}
Oct 31 07:37:29.788989 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:37:29.788956    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:37:30.688669 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.688645    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9kz"
Oct 31 07:37:30.741636 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.741607    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\" (UniqueName: \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\") pod \"3c2df43b-c794-4d6e-aaee-310981189dbd\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") "
Oct 31 07:37:30.741760 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.741654    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\") pod \"3c2df43b-c794-4d6e-aaee-310981189dbd\" (UID: \"3c2df43b-c794-4d6e-aaee-310981189dbd\") "
Oct 31 07:37:30.743750 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.743717    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f" pod "3c2df43b-c794-4d6e-aaee-310981189dbd" (UID: "3c2df43b-c794-4d6e-aaee-310981189dbd"). InnerVolumeSpecName "kube-api-access-6p69f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:37:30.763789 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.763764    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache" pod "3c2df43b-c794-4d6e-aaee-310981189dbd" (UID: "3c2df43b-c794-4d6e-aaee-310981189dbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:37:30.842998 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.842945    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p69f\" (UniqueName: \"kubernetes.io/projected/3c2df43b-c794-4d6e-aaee-310981189dbd-kube-api-access-6p69f\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:30.842998 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:30.842969    4371 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2df43b-c794-4d6e-aaee-310981189dbd-model-cache\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:31.574215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:31.574131    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9kz" event={"ID":"3c2df43b-c794-4d6e-aaee-310981189dbd","Type":"ContainerDied","Data":"45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608"}
Oct 31 07:37:31.574215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:31.574153    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9kz"
Oct 31 07:37:31.574215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:31.574173    4371 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45494bff3dcb007babf11cd06b5f1b91cce2ae00fd6ed4d5cced441e2750e608"
Oct 31 07:37:32.688627 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:32.688586    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:41.925195 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925159    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-sl7pb"]
Oct 31 07:37:41.925594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925456    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerName="s3-init"
Oct 31 07:37:41.925594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925467    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerName="s3-init"
Oct 31 07:37:41.925594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925480    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerName="download-hf-model"
Oct 31 07:37:41.925594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925488    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerName="download-hf-model"
Oct 31 07:37:41.925594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.925565    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2df43b-c794-4d6e-aaee-310981189dbd" containerName="s3-init"
Oct 31 07:37:41.927599 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.927582    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:41.931644 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.931622    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:37:41.933018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.933000    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:37:41.933111 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.933000    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 07:37:41.933111 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.933044    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 31 07:37:41.933195 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.933178    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4r7k\""
Oct 31 07:37:41.936367 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:41.936348    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sl7pb"]
Oct 31 07:37:42.032821 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.032791    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.032987 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.032842    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5br\" (UniqueName: \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.133147 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.133112    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.133300 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.133164    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5br\" (UniqueName: \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.133488 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.133467    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.142817 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.142795    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5br\" (UniqueName: \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\") pod \"s3-tls-init-custom-sl7pb\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") " pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.252840 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.252814    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:42.521810 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.521673    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sl7pb"]
Oct 31 07:37:42.524456 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:37:42.524428    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeed32ef_11b2_4a38_8acc_20e06f4fcbd2.slice/crio-7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864 WatchSource:0}: Error finding container 7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864: Status 404 returned error can't find the container with id 7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864
Oct 31 07:37:42.615211 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.615183    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerStarted","Data":"198122eb4e2e90cf2017e4ff474731560c34a899e1774d2f3d1e1f4a3fb783d7"}
Oct 31 07:37:42.615357 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:42.615220    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerStarted","Data":"7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864"}
Oct 31 07:37:44.624549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:44.624497    4371 generic.go:356] "Generic (PLEG): container finished" podID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerID="198122eb4e2e90cf2017e4ff474731560c34a899e1774d2f3d1e1f4a3fb783d7" exitCode=0
Oct 31 07:37:44.624915 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:44.624572    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerDied","Data":"198122eb4e2e90cf2017e4ff474731560c34a899e1774d2f3d1e1f4a3fb783d7"}
Oct 31 07:37:45.630165 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:45.630133    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerStarted","Data":"b67a9e0daf451145f5a333e2e64a455ff313dd9fe5929445fd94655a11387c38"}
Oct 31 07:37:45.647033 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:45.646989    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-sl7pb" podStartSLOduration=3.093561523 podStartE2EDuration="4.646977258s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.553415735s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:45.64622092 +0000 UTC m=+508.062351592" watchObservedRunningTime="2026-10-31 07:37:45.646977258 +0000 UTC m=+508.063107927"
Oct 31 07:37:49.647477 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:49.647445    4371 generic.go:356] "Generic (PLEG): container finished" podID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerID="b67a9e0daf451145f5a333e2e64a455ff313dd9fe5929445fd94655a11387c38" exitCode=0
Oct 31 07:37:49.647789 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:49.647536    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerDied","Data":"b67a9e0daf451145f5a333e2e64a455ff313dd9fe5929445fd94655a11387c38"}
Oct 31 07:37:50.781369 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:50.781344    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:50.905808 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:50.905741    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\" (UniqueName: \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\") pod \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") "
Oct 31 07:37:50.905808 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:50.905776    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\") pod \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\" (UID: \"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2\") "
Oct 31 07:37:50.907754 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:50.907730    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br" pod "eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" (UID: "eeed32ef-11b2-4a38-8acc-20e06f4fcbd2"). InnerVolumeSpecName "kube-api-access-tw5br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:37:50.928175 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:50.928152    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache" pod "eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" (UID: "eeed32ef-11b2-4a38-8acc-20e06f4fcbd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:37:51.007182 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:51.007155    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5br\" (UniqueName: \"kubernetes.io/projected/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-kube-api-access-tw5br\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:51.007182 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:51.007177    4371 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed32ef-11b2-4a38-8acc-20e06f4fcbd2-model-cache\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:51.656002 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:51.655974    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sl7pb"
Oct 31 07:37:51.656002 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:51.655991    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sl7pb" event={"ID":"eeed32ef-11b2-4a38-8acc-20e06f4fcbd2","Type":"ContainerDied","Data":"7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864"}
Oct 31 07:37:51.656205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:51.656026    4371 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8aad3ba47676581f204b0475e62f73d3a5f04cec059886d1f5cf146351f864"
Oct 31 07:37:52.594679 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:52.594643    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:52.710577 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:52.710544    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:55.342981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.342948    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-5kbwc"]
Oct 31 07:37:55.343384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.343247    4371 cpu_manager.go:414] "containerMap: removing container" podUID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerName="download-hf-model"
Oct 31 07:37:55.343384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.343259    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerName="download-hf-model"
Oct 31 07:37:55.343384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.343267    4371 cpu_manager.go:414] "containerMap: removing container" podUID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerName="s3-tls-init-custom"
Oct 31 07:37:55.343384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.343273    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerName="s3-tls-init-custom"
Oct 31 07:37:55.343384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.343323    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeed32ef-11b2-4a38-8acc-20e06f4fcbd2" containerName="s3-tls-init-custom"
Oct 31 07:37:55.347042 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.347025    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.351785 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.351763    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 07:37:55.353048 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.353024    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-5kbwc"]
Oct 31 07:37:55.353169 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.353143    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 31 07:37:55.353236 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.353210    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4r7k\""
Oct 31 07:37:55.353274 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.353258    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:37:55.353310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.353304    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:37:55.440919 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.440889    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2cv\" (UniqueName: \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.441052 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.440948    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.541769 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.541742    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.541914 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.541790    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2cv\" (UniqueName: \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.542111 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.542091    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.550470 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.550451    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2cv\" (UniqueName: \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\") pod \"s3-tls-init-serving-5kbwc\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") " pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.675301 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.675222    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:37:55.943401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:55.943333    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-5kbwc"]
Oct 31 07:37:55.946581 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:37:55.946555    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod925df15d_1949_4868_875d_0f49de11fd8d.slice/crio-056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af WatchSource:0}: Error finding container 056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af: Status 404 returned error can't find the container with id 056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af
Oct 31 07:37:56.676521 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:56.676470    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerStarted","Data":"a68b918954d3c16cf83cb73b6e9165d5099c472cb49802eb7940616cf289ad1f"}
Oct 31 07:37:56.676910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:56.676532    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerStarted","Data":"056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af"}
Oct 31 07:37:57.681472 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:57.681446    4371 generic.go:356] "Generic (PLEG): container finished" podID="925df15d-1949-4868-875d-0f49de11fd8d" containerID="a68b918954d3c16cf83cb73b6e9165d5099c472cb49802eb7940616cf289ad1f" exitCode=0
Oct 31 07:37:57.681809 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:57.681540    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerDied","Data":"a68b918954d3c16cf83cb73b6e9165d5099c472cb49802eb7940616cf289ad1f"}
Oct 31 07:37:58.686430 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:58.686390    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerStarted","Data":"2942732ecf2b12cf135358f8a65348db313280dfa454bbffce5e3856103297dd"}
Oct 31 07:37:58.703163 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:37:58.703121    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-5kbwc" podStartSLOduration=2.04567687 podStartE2EDuration="3.703109211s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.657432341s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:58.701899299 +0000 UTC m=+521.118029967" watchObservedRunningTime="2026-10-31 07:37:58.703109211 +0000 UTC m=+521.119239883"
Oct 31 07:37:59.788344 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:37:59.788299    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:38:01.542370 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:01.542336    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:04.712622 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:04.712589    4371 generic.go:356] "Generic (PLEG): container finished" podID="925df15d-1949-4868-875d-0f49de11fd8d" containerID="2942732ecf2b12cf135358f8a65348db313280dfa454bbffce5e3856103297dd" exitCode=0
Oct 31 07:38:04.713029 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:04.712662    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerDied","Data":"2942732ecf2b12cf135358f8a65348db313280dfa454bbffce5e3856103297dd"}
Oct 31 07:38:05.853120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:05.853096    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:38:05.926091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:05.926064    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\" (UniqueName: \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\") pod \"925df15d-1949-4868-875d-0f49de11fd8d\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") "
Oct 31 07:38:05.926222 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:05.926099    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\") pod \"925df15d-1949-4868-875d-0f49de11fd8d\" (UID: \"925df15d-1949-4868-875d-0f49de11fd8d\") "
Oct 31 07:38:05.928081 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:05.928054    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv" pod "925df15d-1949-4868-875d-0f49de11fd8d" (UID: "925df15d-1949-4868-875d-0f49de11fd8d"). InnerVolumeSpecName "kube-api-access-6m2cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:38:05.949751 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:05.949723    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache" pod "925df15d-1949-4868-875d-0f49de11fd8d" (UID: "925df15d-1949-4868-875d-0f49de11fd8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:38:06.026697 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:06.026672    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2cv\" (UniqueName: \"kubernetes.io/projected/925df15d-1949-4868-875d-0f49de11fd8d-kube-api-access-6m2cv\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:38:06.026697 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:06.026698    4371 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925df15d-1949-4868-875d-0f49de11fd8d-model-cache\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:38:06.721076 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:06.721043    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-5kbwc"
Oct 31 07:38:06.721076 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:06.721052    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-5kbwc" event={"ID":"925df15d-1949-4868-875d-0f49de11fd8d","Type":"ContainerDied","Data":"056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af"}
Oct 31 07:38:06.721281 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:06.721082    4371 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="056e660fb9600a8181e54860583f4eff7969abdbbe15677d31e966ed86d259af"
Oct 31 07:38:07.229872 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:07.229824    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:07.330371 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:07.330326    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:21.903051 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:21.902955    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:22.009120 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.009084    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:22.935771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.935738    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:22.955557 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955532    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:38:22.955839 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955827    4371 cpu_manager.go:414] "containerMap: removing container" podUID="925df15d-1949-4868-875d-0f49de11fd8d" containerName="download-hf-model"
Oct 31 07:38:22.955887 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955840    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925df15d-1949-4868-875d-0f49de11fd8d" containerName="download-hf-model"
Oct 31 07:38:22.955887 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955851    4371 cpu_manager.go:414] "containerMap: removing container" podUID="925df15d-1949-4868-875d-0f49de11fd8d" containerName="s3-tls-init-serving"
Oct 31 07:38:22.955887 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955857    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925df15d-1949-4868-875d-0f49de11fd8d" containerName="s3-tls-init-serving"
Oct 31 07:38:22.955976 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.955905    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="925df15d-1949-4868-875d-0f49de11fd8d" containerName="s3-tls-init-serving"
Oct 31 07:38:22.960052 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.960037    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:22.968913 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.968891    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:38:22.969342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969313    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:38:22.969445 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969391    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:38:22.969445 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969428    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:38:22.969580 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969466    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Oct 31 07:38:22.969580 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969466    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:38:22.969580 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969490    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 31 07:38:22.969747 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:22.969589    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:38:23.052778 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.052742    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.052939 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.052789    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.052939 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.052865    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.052939 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.052902    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.052939 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.052929    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz252\" (UniqueName: \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.122415 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.122381    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:23.153819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.153788    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154010 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.153824    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154010 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.153847    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz252\" (UniqueName: \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154010 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.153894    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154010 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.153921    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154478 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.154379    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154573 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.154501    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.154684 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.154665    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.156352 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.156323    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.164771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.164750    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz252\" (UniqueName: \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\") pod \"isvc-pmml-v2-kserve-predictor-789b795c74-rxknb\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") " pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.230628 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.230598    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:23.277776 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.277747    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:23.332321 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.332293    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:23.548785 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.548764    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:38:23.550901 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:38:23.550875    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b3b774_6258_436b_a579_509f8aaf6fec.slice/crio-fb0199448435c3ffc808aeafc8ec3da7e707f4d710a84a671e15c845e5e65261 WatchSource:0}: Error finding container fb0199448435c3ffc808aeafc8ec3da7e707f4d710a84a671e15c845e5e65261: Status 404 returned error can't find the container with id fb0199448435c3ffc808aeafc8ec3da7e707f4d710a84a671e15c845e5e65261
Oct 31 07:38:23.782427 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.782384    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerStarted","Data":"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"}
Oct 31 07:38:23.782611 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:23.782461    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerStarted","Data":"fb0199448435c3ffc808aeafc8ec3da7e707f4d710a84a671e15c845e5e65261"}
Oct 31 07:38:27.798889 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:27.798855    4371 generic.go:356] "Generic (PLEG): container finished" podID="08b3b774-6258-436b-a579-509f8aaf6fec" containerID="081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c" exitCode=0
Oct 31 07:38:27.799225 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:27.798930    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerDied","Data":"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"}
Oct 31 07:38:29.800046 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:38:29.800004    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:38:34.230570 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:34.230534    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:35.842938 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:35.842907    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerStarted","Data":"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"}
Oct 31 07:38:37.853731 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:37.853648    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerStarted","Data":"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"}
Oct 31 07:38:37.854091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:37.853778    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:37.873381 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:37.873336    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podStartSLOduration=2.228010249 podStartE2EDuration="15.873323031s" totalImagesPullingTime="9.64236034s" totalInitContainerRuntime="4.002952442s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:37.872983857 +0000 UTC m=+560.289114529" watchObservedRunningTime="2026-10-31 07:38:37.873323031 +0000 UTC m=+560.289453702"
Oct 31 07:38:38.857290 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:38.857251    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:38.858485 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:38.858445    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:38:39.861145 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:39.861097    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:38:44.865685 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:44.865657    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:38:44.866164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:44.866139    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:38:54.866457 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:38:54.866415    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:38:59.789030 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:38:59.788993    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:39:01.585248 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:01.585168    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:04.866568 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:04.866505    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:14.867044 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:14.867005    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:24.866547 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:24.866487    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:29.788715 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:39:29.788681    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:39:34.866173 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:34.866131    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:44.866379 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:44.866339    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:46.061468 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:46.061437    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:46.700003 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:46.699962    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:56.700663 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:39:56.700619    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:39:59.789547 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:39:59.789488    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:40:01.634172 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:01.634142    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:06.701154 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:06.701124    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:40:14.802257 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:14.802224    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:40:14.802660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:14.802480    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" containerID="cri-o://653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5" gracePeriod=30
Oct 31 07:40:14.802660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:14.802552    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kube-rbac-proxy" containerID="cri-o://b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b" gracePeriod=30
Oct 31 07:40:14.862253 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:14.862219    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Oct 31 07:40:14.882165 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:14.882135    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:15.208770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:15.208691    4371 generic.go:356] "Generic (PLEG): container finished" podID="08b3b774-6258-436b-a579-509f8aaf6fec" containerID="b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b" exitCode=0
Oct 31 07:40:15.208770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:15.208761    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerDied","Data":"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"}
Oct 31 07:40:16.700161 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:16.700119    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 31 07:40:18.714418 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.714386    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:18.739838 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.739817    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:40:18.776364 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776337    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\") pod \"08b3b774-6258-436b-a579-509f8aaf6fec\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") "
Oct 31 07:40:18.776528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776418    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\") pod \"08b3b774-6258-436b-a579-509f8aaf6fec\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") "
Oct 31 07:40:18.776528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776469    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\") pod \"08b3b774-6258-436b-a579-509f8aaf6fec\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") "
Oct 31 07:40:18.776528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776489    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\" (UniqueName: \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\") pod \"08b3b774-6258-436b-a579-509f8aaf6fec\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") "
Oct 31 07:40:18.776719 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776632    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\") pod \"08b3b774-6258-436b-a579-509f8aaf6fec\" (UID: \"08b3b774-6258-436b-a579-509f8aaf6fec\") "
Oct 31 07:40:18.776719 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776705    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location" pod "08b3b774-6258-436b-a579-509f8aaf6fec" (UID: "08b3b774-6258-436b-a579-509f8aaf6fec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:40:18.776897 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776806    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config" pod "08b3b774-6258-436b-a579-509f8aaf6fec" (UID: "08b3b774-6258-436b-a579-509f8aaf6fec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:18.776991 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776945    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b3b774-6258-436b-a579-509f8aaf6fec-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:18.776991 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776964    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:18.777105 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.776984    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert" pod "08b3b774-6258-436b-a579-509f8aaf6fec" (UID: "08b3b774-6258-436b-a579-509f8aaf6fec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:18.778714 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.778693    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls" pod "08b3b774-6258-436b-a579-509f8aaf6fec" (UID: "08b3b774-6258-436b-a579-509f8aaf6fec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:40:18.778793 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.778756    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252" pod "08b3b774-6258-436b-a579-509f8aaf6fec" (UID: "08b3b774-6258-436b-a579-509f8aaf6fec"). InnerVolumeSpecName "kube-api-access-xz252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:40:18.877673 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.877620    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b3b774-6258-436b-a579-509f8aaf6fec-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:18.877673 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.877641    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz252\" (UniqueName: \"kubernetes.io/projected/08b3b774-6258-436b-a579-509f8aaf6fec-kube-api-access-xz252\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:18.877673 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:18.877651    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b3b774-6258-436b-a579-509f8aaf6fec-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:19.229026 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.228998    4371 generic.go:356] "Generic (PLEG): container finished" podID="08b3b774-6258-436b-a579-509f8aaf6fec" containerID="653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5" exitCode=0
Oct 31 07:40:19.229178 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.229052    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"
Oct 31 07:40:19.229178 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.229074    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerDied","Data":"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"}
Oct 31 07:40:19.229178 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.229132    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb" event={"ID":"08b3b774-6258-436b-a579-509f8aaf6fec","Type":"ContainerDied","Data":"fb0199448435c3ffc808aeafc8ec3da7e707f4d710a84a671e15c845e5e65261"}
Oct 31 07:40:19.229178 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.229159    4371 scope.go:122] "RemoveContainer" containerID="b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"
Oct 31 07:40:19.237483 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.237463    4371 scope.go:122] "RemoveContainer" containerID="653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"
Oct 31 07:40:19.244530 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.244503    4371 scope.go:122] "RemoveContainer" containerID="081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"
Oct 31 07:40:19.251206 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.251184    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:40:19.252122 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252106    4371 scope.go:122] "RemoveContainer" containerID="b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"
Oct 31 07:40:19.252380 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:40:19.252360    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b\": container with ID starting with b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b not found: ID does not exist" containerID="b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"
Oct 31 07:40:19.252461 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252385    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b"} err="failed to get container status \"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b\": rpc error: code = NotFound desc = could not find container \"b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b\": container with ID starting with b60a41c0b595c51518e72054d4ffa8f112debb5cdc469cc06bcfd79f060c2f6b not found: ID does not exist"
Oct 31 07:40:19.252461 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252403    4371 scope.go:122] "RemoveContainer" containerID="653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"
Oct 31 07:40:19.252661 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:40:19.252643    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5\": container with ID starting with 653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5 not found: ID does not exist" containerID="653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"
Oct 31 07:40:19.252712 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252666    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5"} err="failed to get container status \"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5\": rpc error: code = NotFound desc = could not find container \"653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5\": container with ID starting with 653db553e948ca48848e0cda8ea918e4f39e841f5780ee80a297dd32e54221e5 not found: ID does not exist"
Oct 31 07:40:19.252712 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252680    4371 scope.go:122] "RemoveContainer" containerID="081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"
Oct 31 07:40:19.252903 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:40:19.252887    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c\": container with ID starting with 081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c not found: ID does not exist" containerID="081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"
Oct 31 07:40:19.252945 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.252906    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c"} err="failed to get container status \"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c\": rpc error: code = NotFound desc = could not find container \"081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c\": container with ID starting with 081f679f4684deff15e1aaa874a553ac9911e215b18f707123e1534735e8516c not found: ID does not exist"
Oct 31 07:40:19.256157 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.256137    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-v2-kserve-predictor-789b795c74-rxknb"]
Oct 31 07:40:19.704458 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:19.704371    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" path="/var/lib/kubelet/pods/08b3b774-6258-436b-a579-509f8aaf6fec/volumes"
Oct 31 07:40:20.177141 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:20.177108    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:20.278431 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:20.278400    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:29.788717 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:40:29.788680    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:40:50.050313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:50.050236    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:50.150963 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:50.150933    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:50.437410 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:50.437316    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:50.540217 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:40:50.540185    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:59.788848 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:40:59.788808    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:41:01.710430 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:41:01.710396    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:29.788781 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:41:29.788733    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:41:59.792002 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:41:59.791955    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:42:01.702959 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:01.702863    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:15.656089 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:15.656057    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:15.775240 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:15.775206    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:24.198616 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:24.198573    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:29.789183 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:42:29.789145    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:42:35.841549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:35.841499    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:35.946647 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:35.946617    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:36.324239 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:36.324195    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:42.463837 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:42.463798    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:42.589755 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:42.589720    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:52.563631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:52.563597    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:55.059444 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059408    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059742    4371 cpu_manager.go:414] "containerMap: removing container" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="storage-initializer"
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059754    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="storage-initializer"
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059767    4371 cpu_manager.go:414] "containerMap: removing container" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container"
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059772    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container"
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059789    4371 cpu_manager.go:414] "containerMap: removing container" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kube-rbac-proxy"
Oct 31 07:42:55.059832 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059794    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kube-rbac-proxy"
Oct 31 07:42:55.060017 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059844    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kube-rbac-proxy"
Oct 31 07:42:55.060017 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.059855    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b3b774-6258-436b-a579-509f8aaf6fec" containerName="kserve-container"
Oct 31 07:42:55.062724 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.062708    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.071046 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.071029    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Oct 31 07:42:55.072921 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.072900    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:42:55.073045 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.073030    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Oct 31 07:42:55.074852 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.074837    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:42:55.074928 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.074852    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:42:55.074928 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.074862    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:42:55.074928 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.074871    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:42:55.088333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.087562    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:42:55.123742 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.123706    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:55.152117 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.152085    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.152294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.152129    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.152294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.152151    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.152414 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.152334    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.152414 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.152391    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.230324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.230289    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:55.253193 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253155    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253212    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253237    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253264    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253286    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253708    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253864 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253837    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.253925 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.253908    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.255705 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.255688    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.261490 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.261465    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\") pod \"isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.379632 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.379542    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:42:55.653205 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.653177    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:42:55.655341 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:42:55.655293    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dcc825_ee35_42a3_862d_ef7317a12be2.slice/crio-4554fb40fb9ca68f58754a006a3babfcc530f05cc21b8146315d67b15087b199 WatchSource:0}: Error finding container 4554fb40fb9ca68f58754a006a3babfcc530f05cc21b8146315d67b15087b199: Status 404 returned error can't find the container with id 4554fb40fb9ca68f58754a006a3babfcc530f05cc21b8146315d67b15087b199
Oct 31 07:42:55.785310 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.785269    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerStarted","Data":"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"}
Oct 31 07:42:55.785530 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:55.785339    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerStarted","Data":"4554fb40fb9ca68f58754a006a3babfcc530f05cc21b8146315d67b15087b199"}
Oct 31 07:42:59.793202 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:42:59.793172    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:42:59.802105 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:59.802078    4371 generic.go:356] "Generic (PLEG): container finished" podID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerID="55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00" exitCode=0
Oct 31 07:42:59.802213 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:59.802118    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerDied","Data":"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"}
Oct 31 07:42:59.803387 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:42:59.803370    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:43:01.755917 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:01.755830    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:07.995186 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:07.995152    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:08.129531 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:08.129472    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:12.473365 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:12.473332    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:12.585772 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:12.585734    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:14.762149 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:14.762113    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:14.890385 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:14.890333    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:15.610528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:15.610485    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:22.916104 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.916018    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerStarted","Data":"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"}
Oct 31 07:43:22.916104 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.916089    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerStarted","Data":"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"}
Oct 31 07:43:22.916676 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.916370    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:43:22.916676 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.916418    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:43:22.917699 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.917676    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:22.944036 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:22.943988    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podStartSLOduration=1.068281855 podStartE2EDuration="27.943976117s" totalImagesPullingTime="22.804946022s" totalInitContainerRuntime="4.07074824s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:22.941736911 +0000 UTC m=+845.357867585" watchObservedRunningTime="2026-10-31 07:43:22.943976117 +0000 UTC m=+845.360106788"
Oct 31 07:43:23.920280 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:23.920241    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:28.925234 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:28.925200    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:43:28.925888 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:28.925859    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:29.788781 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:43:29.788738    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:43:38.926623 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:38.926531    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:48.925802 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:48.925758    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:58.926733 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:43:58.926694    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:43:59.789253 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:43:59.789212    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:44:01.795785 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:01.795751    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:08.925902 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:08.925863    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:44:14.479602 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:14.479564    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:15.918230 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:15.918198    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:18.926652 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:18.926611    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:44:28.926666 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:28.926638    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:44:29.790064 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:44:29.790025    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:44:35.885814 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:35.885783    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:44:35.886230 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:35.886017    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" containerID="cri-o://f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5" gracePeriod=30
Oct 31 07:44:35.886230 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:35.886062    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kube-rbac-proxy" containerID="cri-o://0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23" gracePeriod=30
Oct 31 07:44:35.952936 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:35.952910    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:36.186711 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:36.186635    4371 generic.go:356] "Generic (PLEG): container finished" podID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerID="0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23" exitCode=0
Oct 31 07:44:36.186853 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:36.186709    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerDied","Data":"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"}
Oct 31 07:44:38.920862 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:38.920823    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 31 07:44:38.926226 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:38.926202    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Oct 31 07:44:40.624325 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.624301    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:44:40.713019 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.712936    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\") pod \"42dcc825-ee35-42a3-862d-ef7317a12be2\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") "
Oct 31 07:44:40.713019 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713003    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\") pod \"42dcc825-ee35-42a3-862d-ef7317a12be2\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") "
Oct 31 07:44:40.713254 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713171    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\") pod \"42dcc825-ee35-42a3-862d-ef7317a12be2\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") "
Oct 31 07:44:40.713254 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713243    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\") pod \"42dcc825-ee35-42a3-862d-ef7317a12be2\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") "
Oct 31 07:44:40.713368 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713288    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location" pod "42dcc825-ee35-42a3-862d-ef7317a12be2" (UID: "42dcc825-ee35-42a3-862d-ef7317a12be2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:40.713368 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713293    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\") pod \"42dcc825-ee35-42a3-862d-ef7317a12be2\" (UID: \"42dcc825-ee35-42a3-862d-ef7317a12be2\") "
Oct 31 07:44:40.713575 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713547    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config" pod "42dcc825-ee35-42a3-862d-ef7317a12be2" (UID: "42dcc825-ee35-42a3-862d-ef7317a12be2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:40.713646 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713585    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert" pod "42dcc825-ee35-42a3-862d-ef7317a12be2" (UID: "42dcc825-ee35-42a3-862d-ef7317a12be2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:40.713696 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713683    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42dcc825-ee35-42a3-862d-ef7317a12be2-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:40.713755 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713700    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:40.713755 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.713715    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42dcc825-ee35-42a3-862d-ef7317a12be2-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:40.715272 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.715248    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d" pod "42dcc825-ee35-42a3-862d-ef7317a12be2" (UID: "42dcc825-ee35-42a3-862d-ef7317a12be2"). InnerVolumeSpecName "kube-api-access-xkl7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:40.715408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.715392    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls" pod "42dcc825-ee35-42a3-862d-ef7317a12be2" (UID: "42dcc825-ee35-42a3-862d-ef7317a12be2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:40.814480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.814431    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42dcc825-ee35-42a3-862d-ef7317a12be2-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:40.814480 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:40.814474    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/42dcc825-ee35-42a3-862d-ef7317a12be2-kube-api-access-xkl7d\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:41.207403 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.207366    4371 generic.go:356] "Generic (PLEG): container finished" podID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerID="f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5" exitCode=0
Oct 31 07:44:41.207600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.207445    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"
Oct 31 07:44:41.207600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.207454    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerDied","Data":"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"}
Oct 31 07:44:41.207600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.207528    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd" event={"ID":"42dcc825-ee35-42a3-862d-ef7317a12be2","Type":"ContainerDied","Data":"4554fb40fb9ca68f58754a006a3babfcc530f05cc21b8146315d67b15087b199"}
Oct 31 07:44:41.207600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.207558    4371 scope.go:122] "RemoveContainer" containerID="0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"
Oct 31 07:44:41.215934 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.215916    4371 scope.go:122] "RemoveContainer" containerID="f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"
Oct 31 07:44:41.223211 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.223194    4371 scope.go:122] "RemoveContainer" containerID="55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"
Oct 31 07:44:41.229983 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.229960    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:44:41.230341 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.230329    4371 scope.go:122] "RemoveContainer" containerID="0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"
Oct 31 07:44:41.230603 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:44:41.230584    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23\": container with ID starting with 0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23 not found: ID does not exist" containerID="0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"
Oct 31 07:44:41.230653 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.230610    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23"} err="failed to get container status \"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23\": rpc error: code = NotFound desc = could not find container \"0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23\": container with ID starting with 0f69b1d1a0713683b80f9b9f98602d7b4dc51df76b3ef77161c32d3f3a7f3d23 not found: ID does not exist"
Oct 31 07:44:41.230653 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.230625    4371 scope.go:122] "RemoveContainer" containerID="f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"
Oct 31 07:44:41.230857 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:44:41.230840    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5\": container with ID starting with f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5 not found: ID does not exist" containerID="f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"
Oct 31 07:44:41.230906 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.230861    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5"} err="failed to get container status \"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5\": rpc error: code = NotFound desc = could not find container \"f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5\": container with ID starting with f0a446d2106c54ddebaf355f4bbd75d2a8048b10243c0ca79125a85b4d9486d5 not found: ID does not exist"
Oct 31 07:44:41.230906 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.230874    4371 scope.go:122] "RemoveContainer" containerID="55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"
Oct 31 07:44:41.231093 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:44:41.231074    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00\": container with ID starting with 55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00 not found: ID does not exist" containerID="55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"
Oct 31 07:44:41.231143 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.231096    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00"} err="failed to get container status \"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00\": rpc error: code = NotFound desc = could not find container \"55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00\": container with ID starting with 55676479ab37954fd5e34422caa86f9327067494fd4074f27f07b2f49aa75e00 not found: ID does not exist"
Oct 31 07:44:41.233758 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.233738    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-predictor-79bdcb9f6d-27msd"]
Oct 31 07:44:41.704033 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:41.703999    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" path="/var/lib/kubelet/pods/42dcc825-ee35-42a3-862d-ef7317a12be2/volumes"
Oct 31 07:44:43.367630 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367595    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367900    4371 cpu_manager.go:414] "containerMap: removing container" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="storage-initializer"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367913    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="storage-initializer"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367920    4371 cpu_manager.go:414] "containerMap: removing container" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kube-rbac-proxy"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367926    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kube-rbac-proxy"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367950    4371 cpu_manager.go:414] "containerMap: removing container" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.367955    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.368001    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kserve-container"
Oct 31 07:44:43.368016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.368010    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dcc825-ee35-42a3-862d-ef7317a12be2" containerName="kube-rbac-proxy"
Oct 31 07:44:43.372164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.372147    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.376818 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.376794    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:44:43.376952 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.376840    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:44:43.379611 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.379590    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Oct 31 07:44:43.379780 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.379634    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:44:43.380405 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.380037    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:44:43.381224 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.381205    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:44:43.381317 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.381301    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Oct 31 07:44:43.383031 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.383001    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:43.383315 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.383298    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:44:43.432910 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.432865    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjcq\" (UniqueName: \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.433085 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.432992    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.433085 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.433025    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.433085 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.433059    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.433187 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.433118    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.481464 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.481434    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:43.533771 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.533736    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.533889 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.533776    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.533889 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.533805    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.533889 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.533837    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.533889 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.533869    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjcq\" (UniqueName: \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.534356 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.534330    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.534536 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.534494    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.534602 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.534543    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.536177 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.536160    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.541913 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.541893    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjcq\" (UniqueName: \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\") pod \"isvc-predictive-xgboost-predictor-544b584567-b8xsr\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.690808 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.690722    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:43.963042 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:43.963016    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:44:43.965266 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:44:43.965239    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549c8c9b_6cb2_494a_9b19_34f8a2cea03a.slice/crio-b57433157b5100cc846f66e47c7a76df5f06692e24e437edf3a88948784f8f74 WatchSource:0}: Error finding container b57433157b5100cc846f66e47c7a76df5f06692e24e437edf3a88948784f8f74: Status 404 returned error can't find the container with id b57433157b5100cc846f66e47c7a76df5f06692e24e437edf3a88948784f8f74
Oct 31 07:44:44.221943 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:44.221913    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerStarted","Data":"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"}
Oct 31 07:44:44.222107 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:44.221969    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerStarted","Data":"b57433157b5100cc846f66e47c7a76df5f06692e24e437edf3a88948784f8f74"}
Oct 31 07:44:45.427973 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:45.427937    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:45.532385 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:45.532341    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:47.413090 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:47.413058    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:47.513217 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:47.513186    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:48.237144 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:48.237111    4371 generic.go:356] "Generic (PLEG): container finished" podID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerID="21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242" exitCode=0
Oct 31 07:44:48.237306 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:48.237182    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerDied","Data":"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"}
Oct 31 07:44:49.243356 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.243324    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerStarted","Data":"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"}
Oct 31 07:44:49.243798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.243383    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerStarted","Data":"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"}
Oct 31 07:44:49.243798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.243647    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:49.243798 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.243781    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:49.244963 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.244936    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:44:49.264090 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.264050    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podStartSLOduration=2.350311594 podStartE2EDuration="6.264037917s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913726323s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:49.262175591 +0000 UTC m=+931.678306263" watchObservedRunningTime="2026-10-31 07:44:49.264037917 +0000 UTC m=+931.680168589"
Oct 31 07:44:49.740453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.740425    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:49.838266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.838229    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:49.939070 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:49.939031    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:50.247294 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:50.247256    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:44:51.760612 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:51.760583    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:51.858642 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:51.858609    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:51.953680 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:51.953645    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:55.251738 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:55.251706    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:44:55.252362 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:44:55.252329    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:44:59.790294 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:44:59.790255    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:45:00.643441 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:00.643411    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:00.743295 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:00.743261    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:01.839471 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:01.839381    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:05.252685 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:05.252644    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:45:15.252539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:15.252483    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:45:25.252914 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:25.252870    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:45:29.790708 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:45:29.790665    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:45:35.252749 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:35.252708    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:45:45.252726 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:45.252686    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:45:50.008786 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:50.008753    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:55.252703 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:55.252674    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:45:56.811401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:56.811368    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:56.918775 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:45:56.918744    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:59.789195 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:45:59.789159    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:46:01.886237 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:01.886205    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:04.378271 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.378235    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:46:04.378769 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.378530    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" containerID="cri-o://a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591" gracePeriod=30
Oct 31 07:46:04.378769 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.378561    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kube-rbac-proxy" containerID="cri-o://b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be" gracePeriod=30
Oct 31 07:46:04.434098 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.434067    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:04.526462 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.526438    4371 generic.go:356] "Generic (PLEG): container finished" podID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerID="b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be" exitCode=0
Oct 31 07:46:04.526600 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:04.526479    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerDied","Data":"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"}
Oct 31 07:46:05.248248 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:05.248205    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Oct 31 07:46:05.252433 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:05.252408    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Oct 31 07:46:08.812933 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.812911    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:46:08.890096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890030    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\") pod \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") "
Oct 31 07:46:08.890096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890064    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\") pod \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") "
Oct 31 07:46:08.890301 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890101    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\") pod \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") "
Oct 31 07:46:08.890301 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890156    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\" (UniqueName: \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\") pod \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") "
Oct 31 07:46:08.890301 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890190    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\") pod \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\" (UID: \"549c8c9b-6cb2-494a-9b19-34f8a2cea03a\") "
Oct 31 07:46:08.890452 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890348    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location" pod "549c8c9b-6cb2-494a-9b19-34f8a2cea03a" (UID: "549c8c9b-6cb2-494a-9b19-34f8a2cea03a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:08.890452 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890435    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert" pod "549c8c9b-6cb2-494a-9b19-34f8a2cea03a" (UID: "549c8c9b-6cb2-494a-9b19-34f8a2cea03a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:08.890649 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.890624    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config" pod "549c8c9b-6cb2-494a-9b19-34f8a2cea03a" (UID: "549c8c9b-6cb2-494a-9b19-34f8a2cea03a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:08.892103 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.892084    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq" pod "549c8c9b-6cb2-494a-9b19-34f8a2cea03a" (UID: "549c8c9b-6cb2-494a-9b19-34f8a2cea03a"). InnerVolumeSpecName "kube-api-access-mbjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:08.892187 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.892123    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls" pod "549c8c9b-6cb2-494a-9b19-34f8a2cea03a" (UID: "549c8c9b-6cb2-494a-9b19-34f8a2cea03a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:08.991624 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.991603    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:08.991624 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.991625    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:08.991768 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.991635    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:08.991768 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.991646    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjcq\" (UniqueName: \"kubernetes.io/projected/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-api-access-mbjcq\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:08.991768 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:08.991655    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549c8c9b-6cb2-494a-9b19-34f8a2cea03a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:09.548320 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.548288    4371 generic.go:356] "Generic (PLEG): container finished" podID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerID="a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591" exitCode=0
Oct 31 07:46:09.548504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.548350    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"
Oct 31 07:46:09.548504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.548370    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerDied","Data":"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"}
Oct 31 07:46:09.548504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.548438    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr" event={"ID":"549c8c9b-6cb2-494a-9b19-34f8a2cea03a","Type":"ContainerDied","Data":"b57433157b5100cc846f66e47c7a76df5f06692e24e437edf3a88948784f8f74"}
Oct 31 07:46:09.548504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.548474    4371 scope.go:122] "RemoveContainer" containerID="b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"
Oct 31 07:46:09.557197 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.557178    4371 scope.go:122] "RemoveContainer" containerID="a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"
Oct 31 07:46:09.563986 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.563968    4371 scope.go:122] "RemoveContainer" containerID="21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"
Oct 31 07:46:09.570237 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.570218    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:46:09.570801 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.570786    4371 scope.go:122] "RemoveContainer" containerID="b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"
Oct 31 07:46:09.571043 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:09.571026    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be\": container with ID starting with b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be not found: ID does not exist" containerID="b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"
Oct 31 07:46:09.571094 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.571052    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be"} err="failed to get container status \"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be\": rpc error: code = NotFound desc = could not find container \"b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be\": container with ID starting with b97b3931fed5e2df05bc3e90284e9e1ee0f2f93d81c07b8b19e3812c132609be not found: ID does not exist"
Oct 31 07:46:09.571094 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.571068    4371 scope.go:122] "RemoveContainer" containerID="a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"
Oct 31 07:46:09.571292 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:09.571274    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591\": container with ID starting with a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591 not found: ID does not exist" containerID="a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"
Oct 31 07:46:09.571333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.571300    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591"} err="failed to get container status \"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591\": rpc error: code = NotFound desc = could not find container \"a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591\": container with ID starting with a8acae6324c3172553b3b9d588da908d2766408f6ab376cf258351d82e136591 not found: ID does not exist"
Oct 31 07:46:09.571333 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.571314    4371 scope.go:122] "RemoveContainer" containerID="21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"
Oct 31 07:46:09.571535 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:09.571519    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242\": container with ID starting with 21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242 not found: ID does not exist" containerID="21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"
Oct 31 07:46:09.571576 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.571540    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242"} err="failed to get container status \"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242\": rpc error: code = NotFound desc = could not find container \"21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242\": container with ID starting with 21a13f752343d923e4d35e6d5de69e2718569a8bf17e7215ae2da9fa3df90242 not found: ID does not exist"
Oct 31 07:46:09.574189 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.574170    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-predictor-544b584567-b8xsr"]
Oct 31 07:46:09.703153 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:09.703122    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" path="/var/lib/kubelet/pods/549c8c9b-6cb2-494a-9b19-34f8a2cea03a/volumes"
Oct 31 07:46:11.876088 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.876053    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:11.879328 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879303    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:11.879631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879619    4371 cpu_manager.go:414] "containerMap: removing container" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kube-rbac-proxy"
Oct 31 07:46:11.879674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879634    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kube-rbac-proxy"
Oct 31 07:46:11.879674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879650    4371 cpu_manager.go:414] "containerMap: removing container" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="storage-initializer"
Oct 31 07:46:11.879674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879655    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="storage-initializer"
Oct 31 07:46:11.879674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879671    4371 cpu_manager.go:414] "containerMap: removing container" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container"
Oct 31 07:46:11.879802 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879679    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container"
Oct 31 07:46:11.879802 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879752    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kserve-container"
Oct 31 07:46:11.879802 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.879766    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="549c8c9b-6cb2-494a-9b19-34f8a2cea03a" containerName="kube-rbac-proxy"
Oct 31 07:46:11.884257 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.884241    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:11.892678 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.892659    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:46:11.893008 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.892986    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:46:11.893186 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.892986    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:46:11.893266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.893022    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Oct 31 07:46:11.893266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.893028    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Oct 31 07:46:11.893266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.893028    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:46:11.893266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.893054    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:46:11.894343 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:11.894326    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:12.012576 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.012545    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkflp\" (UniqueName: \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.012576 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.012577    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.012829 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.012613    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.012829 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.012726    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.012829 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.012784    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.020892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.020868    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:12.113124 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113094    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113269 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113149    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113269 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113190    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113269 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113227    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkflp\" (UniqueName: \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113269 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113251    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113741 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113711    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113861 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113843    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.113926 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.113909    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.115680 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.115661    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.121626 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.121609    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkflp\" (UniqueName: \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\") pod \"isvc-predictive-lightgbm-predictor-7994568f8b-8whvv\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.201961 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.201905    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:12.471880 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.471858    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:12.473338 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:46:12.473308    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e205272_d1f0_404c_8c29_cad579196c67.slice/crio-0ab405daae3b4761e62cb846c44da00d7028bca46ced0f0378650f860cecb16d WatchSource:0}: Error finding container 0ab405daae3b4761e62cb846c44da00d7028bca46ced0f0378650f860cecb16d: Status 404 returned error can't find the container with id 0ab405daae3b4761e62cb846c44da00d7028bca46ced0f0378650f860cecb16d
Oct 31 07:46:12.561587 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.561551    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerStarted","Data":"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"}
Oct 31 07:46:12.561697 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:12.561630    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerStarted","Data":"0ab405daae3b4761e62cb846c44da00d7028bca46ced0f0378650f860cecb16d"}
Oct 31 07:46:16.579436 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:16.579399    4371 generic.go:356] "Generic (PLEG): container finished" podID="1e205272-d1f0-404c-8c29-cad579196c67" containerID="25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35" exitCode=0
Oct 31 07:46:16.579923 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:16.579478    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerDied","Data":"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"}
Oct 31 07:46:17.587025 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.586992    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerStarted","Data":"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"}
Oct 31 07:46:17.587433 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.587054    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerStarted","Data":"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"}
Oct 31 07:46:17.587433 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.587356    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:17.587558 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.587540    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:17.588744 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.588718    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Oct 31 07:46:17.610242 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:17.610185    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" podStartSLOduration=2.672677256 podStartE2EDuration="6.610167307s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.937490051s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:17.607957538 +0000 UTC m=+1020.024088210" watchObservedRunningTime="2026-10-31 07:46:17.610167307 +0000 UTC m=+1020.026297981"
Oct 31 07:46:18.591258 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:18.591210    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Oct 31 07:46:23.596177 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:23.596146    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:23.597171 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:23.597152    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:27.042396 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:27.042363    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:27.136697 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:27.136653    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:27.881068 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:27.881021    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:27.975741 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:27.975706    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:29.789479 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:29.789442    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:46:29.839645 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:29.839613    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:29.933050 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:29.933017    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:31.192244 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:31.192204    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:31.298829 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:31.298798    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:32.291880 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.291809    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:32.292262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.292145    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container" containerID="cri-o://01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d" gracePeriod=30
Oct 31 07:46:32.292262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.292172    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kube-rbac-proxy" containerID="cri-o://ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87" gracePeriod=30
Oct 31 07:46:32.345799 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.345773    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:32.648891 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.648810    4371 generic.go:356] "Generic (PLEG): container finished" podID="1e205272-d1f0-404c-8c29-cad579196c67" containerID="ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87" exitCode=0
Oct 31 07:46:32.648891 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.648877    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerDied","Data":"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"}
Oct 31 07:46:32.929922 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:32.929891    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:33.087406 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087371    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\" (UniqueName: \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\") pod \"1e205272-d1f0-404c-8c29-cad579196c67\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") "
Oct 31 07:46:33.087585 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087435    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\") pod \"1e205272-d1f0-404c-8c29-cad579196c67\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") "
Oct 31 07:46:33.087585 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087456    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\") pod \"1e205272-d1f0-404c-8c29-cad579196c67\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") "
Oct 31 07:46:33.087585 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087503    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\") pod \"1e205272-d1f0-404c-8c29-cad579196c67\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") "
Oct 31 07:46:33.087585 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087538    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\") pod \"1e205272-d1f0-404c-8c29-cad579196c67\" (UID: \"1e205272-d1f0-404c-8c29-cad579196c67\") "
Oct 31 07:46:33.087892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087870    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert" pod "1e205272-d1f0-404c-8c29-cad579196c67" (UID: "1e205272-d1f0-404c-8c29-cad579196c67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:33.087970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087894    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location" pod "1e205272-d1f0-404c-8c29-cad579196c67" (UID: "1e205272-d1f0-404c-8c29-cad579196c67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:33.087970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.087922    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config" pod "1e205272-d1f0-404c-8c29-cad579196c67" (UID: "1e205272-d1f0-404c-8c29-cad579196c67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:33.089438 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.089417    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls" pod "1e205272-d1f0-404c-8c29-cad579196c67" (UID: "1e205272-d1f0-404c-8c29-cad579196c67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:33.089528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.089463    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp" pod "1e205272-d1f0-404c-8c29-cad579196c67" (UID: "1e205272-d1f0-404c-8c29-cad579196c67"). InnerVolumeSpecName "kube-api-access-fkflp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:33.188296 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.188234    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkflp\" (UniqueName: \"kubernetes.io/projected/1e205272-d1f0-404c-8c29-cad579196c67-kube-api-access-fkflp\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:33.188296 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.188258    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e205272-d1f0-404c-8c29-cad579196c67-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:33.188296 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.188269    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:33.188296 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.188278    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e205272-d1f0-404c-8c29-cad579196c67-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:33.188296 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.188286    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e205272-d1f0-404c-8c29-cad579196c67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:33.461957 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.461852    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:33.551015 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.550989    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:33.655379 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.655349    4371 generic.go:356] "Generic (PLEG): container finished" podID="1e205272-d1f0-404c-8c29-cad579196c67" containerID="01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d" exitCode=0
Oct 31 07:46:33.655549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.655383    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerDied","Data":"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"}
Oct 31 07:46:33.655549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.655430    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"
Oct 31 07:46:33.655549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.655452    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv" event={"ID":"1e205272-d1f0-404c-8c29-cad579196c67","Type":"ContainerDied","Data":"0ab405daae3b4761e62cb846c44da00d7028bca46ced0f0378650f860cecb16d"}
Oct 31 07:46:33.655549 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.655488    4371 scope.go:122] "RemoveContainer" containerID="ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"
Oct 31 07:46:33.663386 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.663372    4371 scope.go:122] "RemoveContainer" containerID="01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"
Oct 31 07:46:33.671262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.671246    4371 scope.go:122] "RemoveContainer" containerID="25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"
Oct 31 07:46:33.678715 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.678692    4371 scope.go:122] "RemoveContainer" containerID="ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"
Oct 31 07:46:33.678954 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:33.678933    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87\": container with ID starting with ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87 not found: ID does not exist" containerID="ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"
Oct 31 07:46:33.679036 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.678967    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87"} err="failed to get container status \"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87\": rpc error: code = NotFound desc = could not find container \"ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87\": container with ID starting with ceb55794664b9db9742fe5d7e31ef9eca9fb5a1ea9d13807707bb57152499b87 not found: ID does not exist"
Oct 31 07:46:33.679036 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.678993    4371 scope.go:122] "RemoveContainer" containerID="01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"
Oct 31 07:46:33.679233 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:33.679217    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d\": container with ID starting with 01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d not found: ID does not exist" containerID="01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"
Oct 31 07:46:33.679290 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.679241    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d"} err="failed to get container status \"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d\": rpc error: code = NotFound desc = could not find container \"01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d\": container with ID starting with 01f04b0d7cf0f5c5deec3c7009776e21609986853d11266f0e4055edc36e747d not found: ID does not exist"
Oct 31 07:46:33.679290 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.679265    4371 scope.go:122] "RemoveContainer" containerID="25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"
Oct 31 07:46:33.679476 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:33.679459    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35\": container with ID starting with 25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35 not found: ID does not exist" containerID="25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"
Oct 31 07:46:33.679538 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.679483    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35"} err="failed to get container status \"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35\": rpc error: code = NotFound desc = could not find container \"25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35\": container with ID starting with 25583e2f697570a9b9cc2b2ce3844fbd239f1ded90d786cd80d1f788d0e51e35 not found: ID does not exist"
Oct 31 07:46:33.680133 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.680108    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:33.682441 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.682422    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-predictor-7994568f8b-8whvv"]
Oct 31 07:46:33.703188 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:33.703168    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e205272-d1f0-404c-8c29-cad579196c67" path="/var/lib/kubelet/pods/1e205272-d1f0-404c-8c29-cad579196c67/volumes"
Oct 31 07:46:34.288416 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:34.288383    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:35.635449 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:35.635414    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:35.922209 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:35.922122    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:43.077648 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:43.077609    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:43.177397 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:46:43.177364    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:59.788924 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:46:59.788886    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:47:01.928137 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:01.928104    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:22.407961 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:22.407927    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:22.504803 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:22.504764    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:29.790052 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:47:29.790014    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:47:53.285907 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:53.285867    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:53.378843 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:53.378808    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:55.371985 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:55.371920    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:55.469024 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:55.468992    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:57.693375 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:57.693345    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:57.879108 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:57.879076    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:59.790439 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:47:59.790406    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:47:59.911602 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:47:59.911572    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:00.007128 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:00.007099    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:00.460842 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:00.460812    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:00.557493 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:00.557459    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:00.788127 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:00.788095    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:01.973434 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:01.973335    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:16.108881 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:16.108835    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:23.524143 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524105    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524409    4371 cpu_manager.go:414] "containerMap: removing container" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kube-rbac-proxy"
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524422    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kube-rbac-proxy"
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524444    4371 cpu_manager.go:414] "containerMap: removing container" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container"
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524449    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container"
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524463    4371 cpu_manager.go:414] "containerMap: removing container" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="storage-initializer"
Oct 31 07:48:23.524525 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524468    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="storage-initializer"
Oct 31 07:48:23.524717 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524548    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kube-rbac-proxy"
Oct 31 07:48:23.524717 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.524564    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e205272-d1f0-404c-8c29-cad579196c67" containerName="kserve-container"
Oct 31 07:48:23.527881 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.527865    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.531232 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.531177    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:23.533063 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.533044    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:48:23.537539 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.537519    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:48:23.537652 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.537539    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 31 07:48:23.537652 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.537563    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:48:23.537652 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.537572    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:48:23.539078 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.539061    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:48:23.629630 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.629601    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff75j\" (UniqueName: \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.629770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.629635    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.629770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.629654    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.629770 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.629716    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.629877 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.629854    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.633208 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.633191    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:48:23.633281 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.633216    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 31 07:48:23.670614 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.670586    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:23.730622 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.730597    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff75j\" (UniqueName: \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.730720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.730624    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.730720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.730642    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.730720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.730659    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.730820 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.730721    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.731036 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.731017    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.731359 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.731331    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.731359 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.731334    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.732964 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.732947    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.738109 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.738089    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff75j\" (UniqueName: \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\") pod \"isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") " pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:23.846018 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:23.845951    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:24.115851 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:24.115826    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:48:24.117974 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:48:24.117948    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d6232d_d93a_49ab_a15c_263f0dc99dcd.slice/crio-a2f0fb9d9f49d6abe67f938ccd325f159fbb7739b5801a4063e3f03d22879b6c WatchSource:0}: Error finding container a2f0fb9d9f49d6abe67f938ccd325f159fbb7739b5801a4063e3f03d22879b6c: Status 404 returned error can't find the container with id a2f0fb9d9f49d6abe67f938ccd325f159fbb7739b5801a4063e3f03d22879b6c
Oct 31 07:48:25.050394 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:25.050357    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerStarted","Data":"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"}
Oct 31 07:48:25.050819 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:25.050420    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerStarted","Data":"a2f0fb9d9f49d6abe67f938ccd325f159fbb7739b5801a4063e3f03d22879b6c"}
Oct 31 07:48:29.064415 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:29.064382    4371 generic.go:356] "Generic (PLEG): container finished" podID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerID="feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7" exitCode=0
Oct 31 07:48:29.064876 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:29.064460    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerDied","Data":"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"}
Oct 31 07:48:29.788874 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:48:29.788841    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:48:30.069986 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:30.069887    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerStarted","Data":"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"}
Oct 31 07:48:30.069986 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:30.069961    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerStarted","Data":"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"}
Oct 31 07:48:30.070567 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:30.070177    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:30.092814 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:30.092770    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podStartSLOduration=3.103723399 podStartE2EDuration="7.092759228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.989035829s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:48:30.091538548 +0000 UTC m=+1152.507669218" watchObservedRunningTime="2026-10-31 07:48:30.092759228 +0000 UTC m=+1152.508889972"
Oct 31 07:48:31.073966 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:31.073932    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:37.082631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:37.082601    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:48:48.268145 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:48.268109    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:48.378521 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:48.378484    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:56.376161 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:48:56.376131    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:59.788613 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:48:59.788580    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:49:02.008624 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:02.008590    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:07.084284 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:07.084245    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.27:8080: connect: connection refused"
Oct 31 07:49:17.083662 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:17.083621    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.27:8080: connect: connection refused"
Oct 31 07:49:18.966868 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:18.966821    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:19.922281 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:19.922233    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:21.571665 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:21.571629    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:21.672293 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:21.672264    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:23.782655 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:23.782622    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:23.874377 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:23.874330    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:24.666826 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:24.666776    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:27.084023 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:27.083982    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.27:8080: connect: connection refused"
Oct 31 07:49:29.789481 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:49:29.789441    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:49:34.161248 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:34.161172    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:34.266096 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:34.266068    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:37.633892 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:37.633849    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:49:43.964865 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:43.964833    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:49:43.965326 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:43.965052    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container" containerID="cri-o://01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea" gracePeriod=30
Oct 31 07:49:43.965326 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:43.965129    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kube-rbac-proxy" containerID="cri-o://cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7" gracePeriod=30
Oct 31 07:49:44.018582 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:44.018551    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:44.337398 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:44.337363    4371 generic.go:356] "Generic (PLEG): container finished" podID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerID="cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7" exitCode=0
Oct 31 07:49:44.337589 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:44.337434    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerDied","Data":"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"}
Oct 31 07:49:47.078442 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:47.078401    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Oct 31 07:49:47.083796 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:47.083771    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.27:8080: connect: connection refused"
Oct 31 07:49:48.615304 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.615279    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:49:48.678912 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.678883    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\") pod \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") "
Oct 31 07:49:48.679072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.678946    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\") pod \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") "
Oct 31 07:49:48.679072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.678975    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\") pod \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") "
Oct 31 07:49:48.679072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.679008    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\" (UniqueName: \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\") pod \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") "
Oct 31 07:49:48.679072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.679054    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\") pod \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\" (UID: \"57d6232d-d93a-49ab-a15c-263f0dc99dcd\") "
Oct 31 07:49:48.679299 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.679256    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config" pod "57d6232d-d93a-49ab-a15c-263f0dc99dcd" (UID: "57d6232d-d93a-49ab-a15c-263f0dc99dcd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:49:48.679355 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.679310    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location" pod "57d6232d-d93a-49ab-a15c-263f0dc99dcd" (UID: "57d6232d-d93a-49ab-a15c-263f0dc99dcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:48.679411 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.679362    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert" pod "57d6232d-d93a-49ab-a15c-263f0dc99dcd" (UID: "57d6232d-d93a-49ab-a15c-263f0dc99dcd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:49:48.681133 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.681109    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls" pod "57d6232d-d93a-49ab-a15c-263f0dc99dcd" (UID: "57d6232d-d93a-49ab-a15c-263f0dc99dcd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:49:48.681236 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.681160    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j" pod "57d6232d-d93a-49ab-a15c-263f0dc99dcd" (UID: "57d6232d-d93a-49ab-a15c-263f0dc99dcd"). InnerVolumeSpecName "kube-api-access-ff75j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:49:48.780091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.780058    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:48.780091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.780087    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:48.780091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.780097    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff75j\" (UniqueName: \"kubernetes.io/projected/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-api-access-ff75j\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:48.780316 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.780107    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57d6232d-d93a-49ab-a15c-263f0dc99dcd-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:48.780316 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:48.780115    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57d6232d-d93a-49ab-a15c-263f0dc99dcd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:49.356947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.356909    4371 generic.go:356] "Generic (PLEG): container finished" podID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerID="01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea" exitCode=0
Oct 31 07:49:49.357155 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.356982    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"
Oct 31 07:49:49.357155 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.356990    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerDied","Data":"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"}
Oct 31 07:49:49.357155 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.357048    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4" event={"ID":"57d6232d-d93a-49ab-a15c-263f0dc99dcd","Type":"ContainerDied","Data":"a2f0fb9d9f49d6abe67f938ccd325f159fbb7739b5801a4063e3f03d22879b6c"}
Oct 31 07:49:49.357155 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.357074    4371 scope.go:122] "RemoveContainer" containerID="cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"
Oct 31 07:49:49.365941 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.365915    4371 scope.go:122] "RemoveContainer" containerID="01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"
Oct 31 07:49:49.372693 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.372677    4371 scope.go:122] "RemoveContainer" containerID="feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"
Oct 31 07:49:49.381186 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.381170    4371 scope.go:122] "RemoveContainer" containerID="cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"
Oct 31 07:49:49.381472 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:49:49.381448    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7\": container with ID starting with cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7 not found: ID does not exist" containerID="cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"
Oct 31 07:49:49.381570 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.381485    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7"} err="failed to get container status \"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7\": rpc error: code = NotFound desc = could not find container \"cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7\": container with ID starting with cee4e67dd97c22e71b40696c15ac618f0a992deb52b75f65337c4af48bbf05a7 not found: ID does not exist"
Oct 31 07:49:49.381570 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.381502    4371 scope.go:122] "RemoveContainer" containerID="01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"
Oct 31 07:49:49.381855 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:49:49.381837    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea\": container with ID starting with 01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea not found: ID does not exist" containerID="01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"
Oct 31 07:49:49.381899 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.381864    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea"} err="failed to get container status \"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea\": rpc error: code = NotFound desc = could not find container \"01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea\": container with ID starting with 01940aea9a9e1df64cdf32fb2ae814bffefd7c9c08386948f531277a103f01ea not found: ID does not exist"
Oct 31 07:49:49.381899 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.381885    4371 scope.go:122] "RemoveContainer" containerID="feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"
Oct 31 07:49:49.382341 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:49:49.382321    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7\": container with ID starting with feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7 not found: ID does not exist" containerID="feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"
Oct 31 07:49:49.382427 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.382344    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7"} err="failed to get container status \"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7\": rpc error: code = NotFound desc = could not find container \"feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7\": container with ID starting with feaebd85da5f25a04d67033f3fadefa786b4701a32f1744282a8c8f15c65d0e7 not found: ID does not exist"
Oct 31 07:49:49.382766 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.382748    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:49:49.383849 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.383833    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-xgboost-v2-predictor-b799569fc-h9fg4"]
Oct 31 07:49:49.703870 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:49.703793    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" path="/var/lib/kubelet/pods/57d6232d-d93a-49ab-a15c-263f0dc99dcd/volumes"
Oct 31 07:49:51.361324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361296    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361587    4371 cpu_manager.go:414] "containerMap: removing container" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="storage-initializer"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361599    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="storage-initializer"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361620    4371 cpu_manager.go:414] "containerMap: removing container" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kube-rbac-proxy"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361625    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kube-rbac-proxy"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361639    4371 cpu_manager.go:414] "containerMap: removing container" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361645    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361693    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kube-rbac-proxy"
Oct 31 07:49:51.361729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.361702    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d6232d-d93a-49ab-a15c-263f0dc99dcd" containerName="kserve-container"
Oct 31 07:49:51.366391 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.366371    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.373645 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.373616    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:49:51.374019 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.373918    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:49:51.374019 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.373988    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:49:51.374458 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.374437    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:51.375086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.375069    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:49:51.375158 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.375119    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 31 07:49:51.375158 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.375123    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 31 07:49:51.375158 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.375133    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:49:51.376383 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.376358    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:49:51.399381 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.399359    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.399491 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.399386    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.399491 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.399410    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.399666 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.399501    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8zj\" (UniqueName: \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.399666 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.399612    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.479834 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.479800    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:51.500401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500372    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8zj\" (UniqueName: \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.500487 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500417    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.500487 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500469    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.500629 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500495    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.500629 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500546    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.500886 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.500867    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.501181 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.501160    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.501262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.501181    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.502962 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.502944    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.508334 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.508308    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8zj\" (UniqueName: \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\") pod \"isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") " pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.684573 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.684475    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:51.958116 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:51.958091    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:49:51.960347 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:49:51.960313    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a928f_4701_4d2b_900d_61e4e57c6501.slice/crio-292e2f579ab1cd1e076c80db06a38cb9490a6539349c708a36c0ce367e3a379e WatchSource:0}: Error finding container 292e2f579ab1cd1e076c80db06a38cb9490a6539349c708a36c0ce367e3a379e: Status 404 returned error can't find the container with id 292e2f579ab1cd1e076c80db06a38cb9490a6539349c708a36c0ce367e3a379e
Oct 31 07:49:52.371082 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:52.371050    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerStarted","Data":"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"}
Oct 31 07:49:52.371445 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:52.371111    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerStarted","Data":"292e2f579ab1cd1e076c80db06a38cb9490a6539349c708a36c0ce367e3a379e"}
Oct 31 07:49:56.386362 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:56.386331    4371 generic.go:356] "Generic (PLEG): container finished" podID="141a928f-4701-4d2b-900d-61e4e57c6501" containerID="8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e" exitCode=0
Oct 31 07:49:56.386738 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:56.386391    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerDied","Data":"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"}
Oct 31 07:49:57.391875 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:57.391843    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerStarted","Data":"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"}
Oct 31 07:49:57.392262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:57.391905    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerStarted","Data":"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"}
Oct 31 07:49:57.392262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:57.392218    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:57.392357 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:57.392266    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:49:57.413919 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:49:57.413877    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" podStartSLOduration=2.279110368 podStartE2EDuration="6.41386394s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.134753572s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:57.411639493 +0000 UTC m=+1239.827770170" watchObservedRunningTime="2026-10-31 07:49:57.41386394 +0000 UTC m=+1239.829994612"
Oct 31 07:49:59.789580 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:49:59.789540    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:50:02.063528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:02.063481    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:03.401758 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:03.401732    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:50:23.086005 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:23.085970    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:23.191035 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:23.191002    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:29.789570 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:29.789531    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:50:33.461179 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:33.461151    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:50:41.672086 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:41.672053    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:50:41.672581 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:41.672359    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kserve-container" containerID="cri-o://a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189" gracePeriod=30
Oct 31 07:50:41.672581 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:41.672438    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kube-rbac-proxy" containerID="cri-o://6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787" gracePeriod=30
Oct 31 07:50:41.739602 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:41.739572    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:42.316016 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.315994    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:50:42.412247 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412168    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\") pod \"141a928f-4701-4d2b-900d-61e4e57c6501\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") "
Oct 31 07:50:42.412247 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412204    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\") pod \"141a928f-4701-4d2b-900d-61e4e57c6501\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") "
Oct 31 07:50:42.412247 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412236    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\") pod \"141a928f-4701-4d2b-900d-61e4e57c6501\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") "
Oct 31 07:50:42.412535 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412285    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\" (UniqueName: \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\") pod \"141a928f-4701-4d2b-900d-61e4e57c6501\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") "
Oct 31 07:50:42.412535 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412334    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\") pod \"141a928f-4701-4d2b-900d-61e4e57c6501\" (UID: \"141a928f-4701-4d2b-900d-61e4e57c6501\") "
Oct 31 07:50:42.412621 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412554    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert" pod "141a928f-4701-4d2b-900d-61e4e57c6501" (UID: "141a928f-4701-4d2b-900d-61e4e57c6501"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:50:42.412674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412584    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location" pod "141a928f-4701-4d2b-900d-61e4e57c6501" (UID: "141a928f-4701-4d2b-900d-61e4e57c6501"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:50:42.412731 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.412709    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config" pod "141a928f-4701-4d2b-900d-61e4e57c6501" (UID: "141a928f-4701-4d2b-900d-61e4e57c6501"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:50:42.414150 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.414124    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls" pod "141a928f-4701-4d2b-900d-61e4e57c6501" (UID: "141a928f-4701-4d2b-900d-61e4e57c6501"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:50:42.414260 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.414173    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj" pod "141a928f-4701-4d2b-900d-61e4e57c6501" (UID: "141a928f-4701-4d2b-900d-61e4e57c6501"). InnerVolumeSpecName "kube-api-access-nc8zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:42.513282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.513247    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc8zj\" (UniqueName: \"kubernetes.io/projected/141a928f-4701-4d2b-900d-61e4e57c6501-kube-api-access-nc8zj\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:42.513282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.513276    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:42.513282 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.513288    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/141a928f-4701-4d2b-900d-61e4e57c6501-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:42.513504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.513299    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141a928f-4701-4d2b-900d-61e4e57c6501-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:42.513504 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.513308    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141a928f-4701-4d2b-900d-61e4e57c6501-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:42.563141 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563110    4371 generic.go:356] "Generic (PLEG): container finished" podID="141a928f-4701-4d2b-900d-61e4e57c6501" containerID="6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787" exitCode=0
Oct 31 07:50:42.563141 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563135    4371 generic.go:356] "Generic (PLEG): container finished" podID="141a928f-4701-4d2b-900d-61e4e57c6501" containerID="a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189" exitCode=0
Oct 31 07:50:42.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563168    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"
Oct 31 07:50:42.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563192    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerDied","Data":"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"}
Oct 31 07:50:42.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563256    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerDied","Data":"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"}
Oct 31 07:50:42.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563279    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk" event={"ID":"141a928f-4701-4d2b-900d-61e4e57c6501","Type":"ContainerDied","Data":"292e2f579ab1cd1e076c80db06a38cb9490a6539349c708a36c0ce367e3a379e"}
Oct 31 07:50:42.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.563302    4371 scope.go:122] "RemoveContainer" containerID="6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"
Oct 31 07:50:42.571528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.571499    4371 scope.go:122] "RemoveContainer" containerID="a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"
Oct 31 07:50:42.578531 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.578500    4371 scope.go:122] "RemoveContainer" containerID="8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"
Oct 31 07:50:42.584714 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.584692    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:50:42.585856 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.585842    4371 scope.go:122] "RemoveContainer" containerID="6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"
Oct 31 07:50:42.586117 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:42.586096    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787\": container with ID starting with 6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787 not found: ID does not exist" containerID="6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"
Oct 31 07:50:42.586171 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586126    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"} err="failed to get container status \"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787\": rpc error: code = NotFound desc = could not find container \"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787\": container with ID starting with 6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787 not found: ID does not exist"
Oct 31 07:50:42.586171 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586143    4371 scope.go:122] "RemoveContainer" containerID="a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"
Oct 31 07:50:42.586361 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:42.586343    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189\": container with ID starting with a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189 not found: ID does not exist" containerID="a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"
Oct 31 07:50:42.586400 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586367    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"} err="failed to get container status \"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189\": rpc error: code = NotFound desc = could not find container \"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189\": container with ID starting with a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189 not found: ID does not exist"
Oct 31 07:50:42.586400 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586383    4371 scope.go:122] "RemoveContainer" containerID="8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"
Oct 31 07:50:42.586674 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:42.586657    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e\": container with ID starting with 8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e not found: ID does not exist" containerID="8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"
Oct 31 07:50:42.586734 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586679    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"} err="failed to get container status \"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e\": rpc error: code = NotFound desc = could not find container \"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e\": container with ID starting with 8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e not found: ID does not exist"
Oct 31 07:50:42.586734 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586692    4371 scope.go:122] "RemoveContainer" containerID="6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"
Oct 31 07:50:42.586941 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586925    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787"} err="failed to get container status \"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787\": rpc error: code = NotFound desc = could not find container \"6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787\": container with ID starting with 6d196947c3aaea302c02b699fa42c7679553f0d224727eb6deee9fe15cc60787 not found: ID does not exist"
Oct 31 07:50:42.587090 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.586941    4371 scope.go:122] "RemoveContainer" containerID="a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"
Oct 31 07:50:42.587203 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.587184    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189"} err="failed to get container status \"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189\": rpc error: code = NotFound desc = could not find container \"a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189\": container with ID starting with a56d90cbb5e62e9b199649cf774985680791ee43860335805b32f032fc6b5189 not found: ID does not exist"
Oct 31 07:50:42.587263 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.587206    4371 scope.go:122] "RemoveContainer" containerID="8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"
Oct 31 07:50:42.587528 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.587488    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e"} err="failed to get container status \"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e\": rpc error: code = NotFound desc = could not find container \"8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e\": container with ID starting with 8f3e4d142ae4e94adb7f1ff900e16912302b7d1b5716b6964581f164d101155e not found: ID does not exist"
Oct 31 07:50:42.589081 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:42.589062    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-lightgbm-v2-predictor-55886ccf46-6g8pk"]
Oct 31 07:50:43.067276 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:43.067245    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:43.193452 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:43.193420    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:43.703247 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:43.703215    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" path="/var/lib/kubelet/pods/141a928f-4701-4d2b-900d-61e4e57c6501/volumes"
Oct 31 07:50:45.195023 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:45.194989    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:46.177180 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177146    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-2c6ebb39"]
Oct 31 07:50:46.177492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177479    4371 cpu_manager.go:414] "containerMap: removing container" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="storage-initializer"
Oct 31 07:50:46.177572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177494    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="storage-initializer"
Oct 31 07:50:46.177572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177528    4371 cpu_manager.go:414] "containerMap: removing container" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kube-rbac-proxy"
Oct 31 07:50:46.177572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177534    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kube-rbac-proxy"
Oct 31 07:50:46.177572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177547    4371 cpu_manager.go:414] "containerMap: removing container" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kserve-container"
Oct 31 07:50:46.177572 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177552    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kserve-container"
Oct 31 07:50:46.177748 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177593    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kube-rbac-proxy"
Oct 31 07:50:46.177748 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.177602    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a928f-4701-4d2b-900d-61e4e57c6501" containerName="kserve-container"
Oct 31 07:50:46.181729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.181711    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:46.185734 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.185703    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:50:46.186204 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.186182    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-2c6ebb39"]
Oct 31 07:50:46.186338 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.186324    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:50:46.186652 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.186640    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4r7k\""
Oct 31 07:50:46.235491 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.235454    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\") pod \"curl-metrics-2c6ebb39\" (UID: \"c5950855-188f-445f-ac75-f944f202ac31\") " pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:46.336168 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.336139    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\") pod \"curl-metrics-2c6ebb39\" (UID: \"c5950855-188f-445f-ac75-f944f202ac31\") " pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:46.344275 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.344253    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\") pod \"curl-metrics-2c6ebb39\" (UID: \"c5950855-188f-445f-ac75-f944f202ac31\") " pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:46.509348 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.509312    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:46.779199 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.779174    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-2c6ebb39"]
Oct 31 07:50:46.781648 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:50:46.781620    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5950855_188f_445f_ac75_f944f202ac31.slice/crio-2ceffd5aec5baa9a9e50ef9b87b16e03d04c63d78da9dcb9bb84e50153fc505c WatchSource:0}: Error finding container 2ceffd5aec5baa9a9e50ef9b87b16e03d04c63d78da9dcb9bb84e50153fc505c: Status 404 returned error can't find the container with id 2ceffd5aec5baa9a9e50ef9b87b16e03d04c63d78da9dcb9bb84e50153fc505c
Oct 31 07:50:46.783540 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:46.783525    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:50:47.585204 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:47.585172    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-2c6ebb39" event={"ID":"c5950855-188f-445f-ac75-f944f202ac31","Type":"ContainerStarted","Data":"2ceffd5aec5baa9a9e50ef9b87b16e03d04c63d78da9dcb9bb84e50153fc505c"}
Oct 31 07:50:48.590417 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:48.590385    4371 generic.go:356] "Generic (PLEG): container finished" podID="c5950855-188f-445f-ac75-f944f202ac31" containerID="99363a5fee5c15558015b29c9bbc702f4246ab54ab1322f21f331168f4e10c48" exitCode=0
Oct 31 07:50:48.590841 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:48.590476    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-2c6ebb39" event={"ID":"c5950855-188f-445f-ac75-f944f202ac31","Type":"ContainerDied","Data":"99363a5fee5c15558015b29c9bbc702f4246ab54ab1322f21f331168f4e10c48"}
Oct 31 07:50:49.711533 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.711498    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:49.741204 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.741182    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-2c6ebb39_c5950855-188f-445f-ac75-f944f202ac31/curl-metrics-2c6ebb39/0.log"
Oct 31 07:50:49.757401 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.757380    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\") pod \"c5950855-188f-445f-ac75-f944f202ac31\" (UID: \"c5950855-188f-445f-ac75-f944f202ac31\") "
Oct 31 07:50:49.759239 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.759212    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42" pod "c5950855-188f-445f-ac75-f944f202ac31" (UID: "c5950855-188f-445f-ac75-f944f202ac31"). InnerVolumeSpecName "kube-api-access-7xt42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:49.807026 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.806995    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-2c6ebb39"]
Oct 31 07:50:49.810871 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.810851    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-2c6ebb39"]
Oct 31 07:50:49.858150 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:49.858093    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/c5950855-188f-445f-ac75-f944f202ac31-kube-api-access-7xt42\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:50.385103 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.385068    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-b99f3cfe"]
Oct 31 07:50:50.385414 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.385401    4371 cpu_manager.go:414] "containerMap: removing container" podUID="c5950855-188f-445f-ac75-f944f202ac31" containerName="curl-metrics-2c6ebb39"
Oct 31 07:50:50.385453 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.385416    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5950855-188f-445f-ac75-f944f202ac31" containerName="curl-metrics-2c6ebb39"
Oct 31 07:50:50.385489 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.385465    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5950855-188f-445f-ac75-f944f202ac31" containerName="curl-metrics-2c6ebb39"
Oct 31 07:50:50.388378 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.388364    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:50.393960 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.393926    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b99f3cfe"]
Oct 31 07:50:50.463069 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.463036    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6k8\" (UniqueName: \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\") pod \"curl-metrics-b99f3cfe\" (UID: \"3539cbae-a51b-4985-ab66-f62a4d345555\") " pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:50.563833 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.563803    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6k8\" (UniqueName: \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\") pod \"curl-metrics-b99f3cfe\" (UID: \"3539cbae-a51b-4985-ab66-f62a4d345555\") " pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:50.572167 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.572142    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6k8\" (UniqueName: \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\") pod \"curl-metrics-b99f3cfe\" (UID: \"3539cbae-a51b-4985-ab66-f62a4d345555\") " pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:50.601105 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.601081    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:50.601105 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.601100    4371 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ceffd5aec5baa9a9e50ef9b87b16e03d04c63d78da9dcb9bb84e50153fc505c"
Oct 31 07:50:50.611472 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:50.611448    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-2c6ebb39\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="c5950855-188f-445f-ac75-f944f202ac31" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:50.700686 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.700615    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:50.966200 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:50.966174    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b99f3cfe"]
Oct 31 07:50:50.968396 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:50:50.968369    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3539cbae_a51b_4985_ab66_f62a4d345555.slice/crio-b4f8cbd5f01c97dbc325cf40807afe4a192d253659967b56918e06476fa0f78d WatchSource:0}: Error finding container b4f8cbd5f01c97dbc325cf40807afe4a192d253659967b56918e06476fa0f78d: Status 404 returned error can't find the container with id b4f8cbd5f01c97dbc325cf40807afe4a192d253659967b56918e06476fa0f78d
Oct 31 07:50:51.607204 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:51.607172    4371 generic.go:356] "Generic (PLEG): container finished" podID="3539cbae-a51b-4985-ab66-f62a4d345555" containerID="e21bee4b6596ce8f05b85ade9a84994681ee47c7a5915276140adee26846b767" exitCode=0
Oct 31 07:50:51.607375 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:51.607211    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b99f3cfe" event={"ID":"3539cbae-a51b-4985-ab66-f62a4d345555","Type":"ContainerDied","Data":"e21bee4b6596ce8f05b85ade9a84994681ee47c7a5915276140adee26846b767"}
Oct 31 07:50:51.607375 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:51.607239    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b99f3cfe" event={"ID":"3539cbae-a51b-4985-ab66-f62a4d345555","Type":"ContainerStarted","Data":"b4f8cbd5f01c97dbc325cf40807afe4a192d253659967b56918e06476fa0f78d"}
Oct 31 07:50:51.624639 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:51.624599    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-2c6ebb39\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="c5950855-188f-445f-ac75-f944f202ac31" pod="kserve/curl-metrics-2c6ebb39"
Oct 31 07:50:51.704324 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:51.704293    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5950855-188f-445f-ac75-f944f202ac31" path="/var/lib/kubelet/pods/c5950855-188f-445f-ac75-f944f202ac31/volumes"
Oct 31 07:50:52.732130 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.732103    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:52.764701 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.764676    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-b99f3cfe_3539cbae-a51b-4985-ab66-f62a4d345555/curl-metrics-b99f3cfe/0.log"
Oct 31 07:50:52.780486 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.780461    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\" (UniqueName: \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\") pod \"3539cbae-a51b-4985-ab66-f62a4d345555\" (UID: \"3539cbae-a51b-4985-ab66-f62a4d345555\") "
Oct 31 07:50:52.782475 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.782446    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8" pod "3539cbae-a51b-4985-ab66-f62a4d345555" (UID: "3539cbae-a51b-4985-ab66-f62a4d345555"). InnerVolumeSpecName "kube-api-access-7j6k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:52.837469 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.837424    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-b99f3cfe"]
Oct 31 07:50:52.839009 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.838989    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-b99f3cfe"]
Oct 31 07:50:52.881254 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:52.881197    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6k8\" (UniqueName: \"kubernetes.io/projected/3539cbae-a51b-4985-ab66-f62a4d345555-kube-api-access-7j6k8\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:53.387108 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:53.387078    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:53.492956 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:53.492925    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:53.617447 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:53.617416    4371 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f8cbd5f01c97dbc325cf40807afe4a192d253659967b56918e06476fa0f78d"
Oct 31 07:50:53.617447 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:53.617424    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:53.627935 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:53.627907    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-b99f3cfe\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="3539cbae-a51b-4985-ab66-f62a4d345555" pod="kserve/curl-metrics-b99f3cfe"
Oct 31 07:50:53.703492 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:53.703422    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3539cbae-a51b-4985-ab66-f62a4d345555" path="/var/lib/kubelet/pods/3539cbae-a51b-4985-ab66-f62a4d345555/volumes"
Oct 31 07:50:54.076531 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:54.076480    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:54.178164 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:54.178131    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:55.941300 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:55.941267    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:56.048929 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:56.048897    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:58.284099 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:58.284067    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:58.387550 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:58.387499    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:58.490195 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:50:58.490162    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:59.789167 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:50:59.789136    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:51:00.519626 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:00.519582    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:00.614477 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:00.614443    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:01.452569 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:01.452536    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:02.099450 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:02.099389    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:09.246757 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:09.246722    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:09.350288 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:51:09.350261    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:29.789315 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:51:29.789273    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:51:59.789424 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:51:59.789354    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:52:02.176532 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:02.176453    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:16.008777 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:16.008735    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:29.788468 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:52:29.788430    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:52:42.209342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:42.209244    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:42.313762 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:42.313733    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:48.466029 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:48.465998    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:54.822998 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:52:54.822958    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:59.788604 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:52:59.788570    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:53:02.139441 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.139412    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:02.145661 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.145638    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:53:02.145924 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.145913    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3539cbae-a51b-4985-ab66-f62a4d345555" containerName="curl-metrics-b99f3cfe"
Oct 31 07:53:02.145970 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.145925    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3539cbae-a51b-4985-ab66-f62a4d345555" containerName="curl-metrics-b99f3cfe"
Oct 31 07:53:02.146053 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.146025    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="3539cbae-a51b-4985-ab66-f62a4d345555" containerName="curl-metrics-b99f3cfe"
Oct 31 07:53:02.149247 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.149230    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.153020 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.153002    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:53:02.153275 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.153257    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 31 07:53:02.153354 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.153295    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:53:02.154594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.154573    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:53:02.154594 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.154591    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:53:02.154757 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.154619    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 31 07:53:02.154757 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.154663    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:53:02.158272 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.158251    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:53:02.240143 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.240116    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:02.250298 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.250277    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnmd\" (UniqueName: \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.250384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.250310    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.250384 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.250331    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.250487 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.250471    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.250563 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.250500    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.342031 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.341982    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:02.351066 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351041    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnmd\" (UniqueName: \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351153 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351078    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351153 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351104    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351155    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351262 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351175    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351745 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351724    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351943 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351923    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.351943 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.351936    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.353328 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.353309    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.358884 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.358836    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnmd\" (UniqueName: \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\") pod \"sklearn-v2-mlserver-predictor-5c876d6ddd-827xh\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") " pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.467473 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.467442    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:53:02.738903 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:02.738882    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:53:02.740897 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:53:02.740868    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e432b0_0f97_405d_9a8c_40458f1d53d5.slice/crio-0eabad61d1064141c2e8925dc3987c736793f9f3bebc5f904caeb63ea86fb796 WatchSource:0}: Error finding container 0eabad61d1064141c2e8925dc3987c736793f9f3bebc5f904caeb63ea86fb796: Status 404 returned error can't find the container with id 0eabad61d1064141c2e8925dc3987c736793f9f3bebc5f904caeb63ea86fb796
Oct 31 07:53:03.088077 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:03.087988    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerStarted","Data":"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"}
Oct 31 07:53:03.088224 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:03.088086    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerStarted","Data":"0eabad61d1064141c2e8925dc3987c736793f9f3bebc5f904caeb63ea86fb796"}
Oct 31 07:53:07.103708 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:07.103671    4371 generic.go:356] "Generic (PLEG): container finished" podID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerID="1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a" exitCode=0
Oct 31 07:53:07.104072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:07.103743    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerDied","Data":"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"}
Oct 31 07:53:12.009383 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:12.009352    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:12.111021 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:12.110969    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:14.107978 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:14.107947    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:14.217731 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:14.217697    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:27.198645 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:27.198591    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:27.301727 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:27.301686    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:29.794746 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:53:29.794703    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:53:34.183987 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:34.183944    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:34.284285 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:34.284247    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:37.472421 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:53:37.472379    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:59.793922 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:53:59.793803    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:54:02.231382 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:54:02.231299    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:29.791941 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:54:29.791893    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:54:59.793016 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:54:59.792977    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:55:02.277854 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:02.277814    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:04.620187 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:04.620151    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerStarted","Data":"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"}
Oct 31 07:55:04.620674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:04.620205    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerStarted","Data":"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"}
Oct 31 07:55:04.620674 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:04.620268    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:05.623584 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:05.623552    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:10.155695 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:10.155659    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:10.251720 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:10.251686    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:11.633380 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:11.633353    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:11.655179 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:11.655133    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" podStartSLOduration=8.823215876999999 podStartE2EDuration="2m9.655119502s" totalImagesPullingTime="1m56.954667055s" totalInitContainerRuntime="3.87723657s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:55:04.640865462 +0000 UTC m=+1547.056996134" watchObservedRunningTime="2026-10-31 07:55:11.655119502 +0000 UTC m=+1554.071250174"
Oct 31 07:55:29.788844 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:55:29.788809    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:55:41.637331 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:41.637242    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 31 07:55:42.042658 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:42.042621    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:42.152748 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:42.152718    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:51.636300 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:51.636273    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:52.656158 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.656108    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:55:52.656644 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.656417    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kserve-container" containerID="cri-o://28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5" gracePeriod=30
Oct 31 07:55:52.656644 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.656459    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kube-rbac-proxy" containerID="cri-o://59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74" gracePeriod=30
Oct 31 07:55:52.724990 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.724965    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:52.799287 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.799258    4371 generic.go:356] "Generic (PLEG): container finished" podID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerID="59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74" exitCode=0
Oct 31 07:55:52.799423 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:52.799330    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerDied","Data":"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"}
Oct 31 07:55:56.627788 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:56.627746    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Oct 31 07:55:59.688070 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.688047    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:59.790853 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:55:59.790768    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:55:59.826776 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.826745    4371 generic.go:356] "Generic (PLEG): container finished" podID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerID="28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5" exitCode=0
Oct 31 07:55:59.826947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.826797    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"
Oct 31 07:55:59.826947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.826794    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerDied","Data":"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"}
Oct 31 07:55:59.826947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.826927    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh" event={"ID":"33e432b0-0f97-405d-9a8c-40458f1d53d5","Type":"ContainerDied","Data":"0eabad61d1064141c2e8925dc3987c736793f9f3bebc5f904caeb63ea86fb796"}
Oct 31 07:55:59.827072 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.826969    4371 scope.go:122] "RemoveContainer" containerID="59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"
Oct 31 07:55:59.834691 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.834671    4371 scope.go:122] "RemoveContainer" containerID="28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"
Oct 31 07:55:59.842013 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.841999    4371 scope.go:122] "RemoveContainer" containerID="1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"
Oct 31 07:55:59.849313 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.849292    4371 scope.go:122] "RemoveContainer" containerID="59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"
Oct 31 07:55:59.849584 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:55:59.849566    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74\": container with ID starting with 59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74 not found: ID does not exist" containerID="59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"
Oct 31 07:55:59.849649 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.849593    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74"} err="failed to get container status \"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74\": rpc error: code = NotFound desc = could not find container \"59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74\": container with ID starting with 59658148dc37b275acb7ae18f68abf1695a777bcfa7c53686c6b287785bd5e74 not found: ID does not exist"
Oct 31 07:55:59.849649 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.849609    4371 scope.go:122] "RemoveContainer" containerID="28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"
Oct 31 07:55:59.849863 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:55:59.849846    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5\": container with ID starting with 28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5 not found: ID does not exist" containerID="28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"
Oct 31 07:55:59.849921 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.849872    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5"} err="failed to get container status \"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5\": rpc error: code = NotFound desc = could not find container \"28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5\": container with ID starting with 28d81ee7f8dbb8164c9211a7b8ada160cc637b007a6f5dae1968e547e3d8f3f5 not found: ID does not exist"
Oct 31 07:55:59.849921 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.849888    4371 scope.go:122] "RemoveContainer" containerID="1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"
Oct 31 07:55:59.850142 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:55:59.850124    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a\": container with ID starting with 1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a not found: ID does not exist" containerID="1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"
Oct 31 07:55:59.850185 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.850145    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a"} err="failed to get container status \"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a\": rpc error: code = NotFound desc = could not find container \"1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a\": container with ID starting with 1000aa3dbcf30dd4d4e63bb56e8fec53360d0281406ceff3b8feba10f016df5a not found: ID does not exist"
Oct 31 07:55:59.858469 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858453    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\" (UniqueName: \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\") pod \"33e432b0-0f97-405d-9a8c-40458f1d53d5\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") "
Oct 31 07:55:59.858550 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858478    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\") pod \"33e432b0-0f97-405d-9a8c-40458f1d53d5\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") "
Oct 31 07:55:59.858550 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858497    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\") pod \"33e432b0-0f97-405d-9a8c-40458f1d53d5\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") "
Oct 31 07:55:59.858657 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858597    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\") pod \"33e432b0-0f97-405d-9a8c-40458f1d53d5\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") "
Oct 31 07:55:59.858657 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858621    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\") pod \"33e432b0-0f97-405d-9a8c-40458f1d53d5\" (UID: \"33e432b0-0f97-405d-9a8c-40458f1d53d5\") "
Oct 31 07:55:59.858949 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858914    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location" pod "33e432b0-0f97-405d-9a8c-40458f1d53d5" (UID: "33e432b0-0f97-405d-9a8c-40458f1d53d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:55:59.858949 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.858941    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config" pod "33e432b0-0f97-405d-9a8c-40458f1d53d5" (UID: "33e432b0-0f97-405d-9a8c-40458f1d53d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:59.859075 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.859012    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert" pod "33e432b0-0f97-405d-9a8c-40458f1d53d5" (UID: "33e432b0-0f97-405d-9a8c-40458f1d53d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:59.861408 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.861383    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd" pod "33e432b0-0f97-405d-9a8c-40458f1d53d5" (UID: "33e432b0-0f97-405d-9a8c-40458f1d53d5"). InnerVolumeSpecName "kube-api-access-tqnmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:59.861488 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.861424    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls" pod "33e432b0-0f97-405d-9a8c-40458f1d53d5" (UID: "33e432b0-0f97-405d-9a8c-40458f1d53d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:55:59.959424 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.959393    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnmd\" (UniqueName: \"kubernetes.io/projected/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-api-access-tqnmd\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:59.959424 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.959418    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e432b0-0f97-405d-9a8c-40458f1d53d5-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:59.959424 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.959428    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:59.959670 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.959438    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e432b0-0f97-405d-9a8c-40458f1d53d5-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:59.959670 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:55:59.959446    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e432b0-0f97-405d-9a8c-40458f1d53d5-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:00.148729 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:00.148701    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:56:00.151207 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:00.151183    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/sklearn-v2-mlserver-predictor-5c876d6ddd-827xh"]
Oct 31 07:56:01.703582 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:01.703549    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" path="/var/lib/kubelet/pods/33e432b0-0f97-405d-9a8c-40458f1d53d5/volumes"
Oct 31 07:56:02.161454 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:02.161415    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:02.278215 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:02.278170    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:02.376444 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:02.376415    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:29.789192 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:56:29.789158    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:56:45.602718 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:45.602682    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:48.935977 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.935945    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936222    4371 cpu_manager.go:414] "containerMap: removing container" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kserve-container"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936233    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kserve-container"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936253    4371 cpu_manager.go:414] "containerMap: removing container" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kube-rbac-proxy"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936258    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kube-rbac-proxy"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936267    4371 cpu_manager.go:414] "containerMap: removing container" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="storage-initializer"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936272    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="storage-initializer"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936316    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kserve-container"
Oct 31 07:56:48.936349 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.936326    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e432b0-0f97-405d-9a8c-40458f1d53d5" containerName="kube-rbac-proxy"
Oct 31 07:56:48.938332 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.938273    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:48.939618 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.939590    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:48.947568 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.947548    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:56:48.947660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.947622    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:56:48.947660 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.947646    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 31 07:56:48.947794 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.947780    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:56:48.949098 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.949077    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:56:48.949197 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.949166    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 31 07:56:48.949253 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.949219    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:56:48.949299 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:48.949267    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:56:49.007598 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.007576    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.007725 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.007609    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.007725 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.007633    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.007846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.007773    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9pj\" (UniqueName: \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.007846 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.007819    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.087365 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.087330    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:49.108947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.108922    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9pj\" (UniqueName: \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109097 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.108965    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109097 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109035    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109207 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109157    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109253 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109205    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109554 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109534    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109702 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109683    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.109764 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.109749    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.111562 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.111546    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.117111 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.117093    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9pj\" (UniqueName: \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\") pod \"isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.254011 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.253987    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:49.525132 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:49.525107    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:56:49.527824 ip-10-0-137-69 kubenswrapper[4371]: W1031 07:56:49.527798    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dfd486_658e_4c6a_9e8e_373868947e94.slice/crio-f510e7b1939776687de74170b1ff6782c728a558f619315dc7d8bd75e1603671 WatchSource:0}: Error finding container f510e7b1939776687de74170b1ff6782c728a558f619315dc7d8bd75e1603671: Status 404 returned error can't find the container with id f510e7b1939776687de74170b1ff6782c728a558f619315dc7d8bd75e1603671
Oct 31 07:56:50.004579 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:50.004544    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerStarted","Data":"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"}
Oct 31 07:56:50.004953 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:50.004610    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerStarted","Data":"f510e7b1939776687de74170b1ff6782c728a558f619315dc7d8bd75e1603671"}
Oct 31 07:56:54.020046 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:54.020009    4371 generic.go:356] "Generic (PLEG): container finished" podID="50dfd486-658e-4c6a-9e8e-373868947e94" containerID="0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db" exitCode=0
Oct 31 07:56:54.020536 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:54.020058    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerDied","Data":"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"}
Oct 31 07:56:55.025639 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:55.025606    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerStarted","Data":"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"}
Oct 31 07:56:55.026054 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:55.025669    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerStarted","Data":"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"}
Oct 31 07:56:55.026054 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:55.025873    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:55.047888 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:55.047846    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" podStartSLOduration=3.211706142 podStartE2EDuration="7.047834684s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836128542s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:56:55.045934563 +0000 UTC m=+1657.462065230" watchObservedRunningTime="2026-10-31 07:56:55.047834684 +0000 UTC m=+1657.463965353"
Oct 31 07:56:56.029526 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:56:56.029472    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:56:59.788580 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:56:59.788541    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:57:02.040129 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:02.040056    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:57:02.373305 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:02.373209    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:17.874417 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:17.874381    4371 scope.go:122] "RemoveContainer" containerID="e21bee4b6596ce8f05b85ade9a84994681ee47c7a5915276140adee26846b767"
Oct 31 07:57:17.881947 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:17.881916    4371 scope.go:122] "RemoveContainer" containerID="99363a5fee5c15558015b29c9bbc702f4246ab54ab1322f21f331168f4e10c48"
Oct 31 07:57:29.789120 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:57:29.789074    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:57:32.131430 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:32.131378    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 31 07:57:42.043411 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:42.043380    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:57:43.656666 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:43.656632    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:49.345527 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:49.345484    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:57:49.345952 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:49.345771    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kserve-container" containerID="cri-o://0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce" gracePeriod=30
Oct 31 07:57:49.345952 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:49.345859    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kube-rbac-proxy" containerID="cri-o://e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4" gracePeriod=30
Oct 31 07:57:49.391569 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:49.391542    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:50.243774 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:50.243740    4371 generic.go:356] "Generic (PLEG): container finished" podID="50dfd486-658e-4c6a-9e8e-373868947e94" containerID="e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4" exitCode=0
Oct 31 07:57:50.243774 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:50.243778    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerDied","Data":"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"}
Oct 31 07:57:52.035342 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:52.035296    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Oct 31 07:57:56.476911 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.476885    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:57:56.501690 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.501661    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\") pod \"50dfd486-658e-4c6a-9e8e-373868947e94\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") "
Oct 31 07:57:56.501805 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.501733    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\" (UniqueName: \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\") pod \"50dfd486-658e-4c6a-9e8e-373868947e94\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") "
Oct 31 07:57:56.501805 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.501772    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\") pod \"50dfd486-658e-4c6a-9e8e-373868947e94\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") "
Oct 31 07:57:56.501922 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.501803    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\") pod \"50dfd486-658e-4c6a-9e8e-373868947e94\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") "
Oct 31 07:57:56.501922 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.501870    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\" (UniqueName: \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\") pod \"50dfd486-658e-4c6a-9e8e-373868947e94\" (UID: \"50dfd486-658e-4c6a-9e8e-373868947e94\") "
Oct 31 07:57:56.502081 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.502053    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert" pod "50dfd486-658e-4c6a-9e8e-373868947e94" (UID: "50dfd486-658e-4c6a-9e8e-373868947e94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:57:56.502210 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.502168    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config" pod "50dfd486-658e-4c6a-9e8e-373868947e94" (UID: "50dfd486-658e-4c6a-9e8e-373868947e94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:57:56.502680 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.502652    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location" pod "50dfd486-658e-4c6a-9e8e-373868947e94" (UID: "50dfd486-658e-4c6a-9e8e-373868947e94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:57:56.504835 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.504813    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls" pod "50dfd486-658e-4c6a-9e8e-373868947e94" (UID: "50dfd486-658e-4c6a-9e8e-373868947e94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:57:56.504931 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.504835    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj" pod "50dfd486-658e-4c6a-9e8e-373868947e94" (UID: "50dfd486-658e-4c6a-9e8e-373868947e94"). InnerVolumeSpecName "kube-api-access-rz9pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:57:56.602981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.602888    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9pj\" (UniqueName: \"kubernetes.io/projected/50dfd486-658e-4c6a-9e8e-373868947e94-kube-api-access-rz9pj\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:57:56.602981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.602919    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50dfd486-658e-4c6a-9e8e-373868947e94-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:57:56.602981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.602932    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:57:56.602981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.602946    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50dfd486-658e-4c6a-9e8e-373868947e94-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:57:56.602981 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:56.602958    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50dfd486-658e-4c6a-9e8e-373868947e94-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:57:57.272089 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.272053    4371 generic.go:356] "Generic (PLEG): container finished" podID="50dfd486-658e-4c6a-9e8e-373868947e94" containerID="0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce" exitCode=0
Oct 31 07:57:57.272266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.272112    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"
Oct 31 07:57:57.272266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.272131    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerDied","Data":"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"}
Oct 31 07:57:57.272266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.272194    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5" event={"ID":"50dfd486-658e-4c6a-9e8e-373868947e94","Type":"ContainerDied","Data":"f510e7b1939776687de74170b1ff6782c728a558f619315dc7d8bd75e1603671"}
Oct 31 07:57:57.272266 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.272221    4371 scope.go:122] "RemoveContainer" containerID="e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"
Oct 31 07:57:57.280531 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.280495    4371 scope.go:122] "RemoveContainer" containerID="0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"
Oct 31 07:57:57.287583 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.287565    4371 scope.go:122] "RemoveContainer" containerID="0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"
Oct 31 07:57:57.293479 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.293460    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:57:57.294621 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.294603    4371 scope.go:122] "RemoveContainer" containerID="e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"
Oct 31 07:57:57.294852 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:57:57.294832    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4\": container with ID starting with e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4 not found: ID does not exist" containerID="e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"
Oct 31 07:57:57.294914 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.294859    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4"} err="failed to get container status \"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4\": rpc error: code = NotFound desc = could not find container \"e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4\": container with ID starting with e71c9291a9c8c9b81ffa30e9a5ef548c74f66467db1d985a3b8187910fc9e3e4 not found: ID does not exist"
Oct 31 07:57:57.294914 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.294875    4371 scope.go:122] "RemoveContainer" containerID="0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"
Oct 31 07:57:57.295114 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:57:57.295099    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce\": container with ID starting with 0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce not found: ID does not exist" containerID="0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"
Oct 31 07:57:57.295161 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.295119    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce"} err="failed to get container status \"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce\": rpc error: code = NotFound desc = could not find container \"0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce\": container with ID starting with 0047954a1422d5e30218e15c3e1b673de12c152785bc84ad4d21e1327b2459ce not found: ID does not exist"
Oct 31 07:57:57.295161 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.295132    4371 scope.go:122] "RemoveContainer" containerID="0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"
Oct 31 07:57:57.295365 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:57:57.295334    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db\": container with ID starting with 0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db not found: ID does not exist" containerID="0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"
Oct 31 07:57:57.295566 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.295371    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db"} err="failed to get container status \"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db\": rpc error: code = NotFound desc = could not find container \"0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db\": container with ID starting with 0a9d9d3c00403bd16df7fcaa4646ec080617e0a39e3cb095e04563d42456c7db not found: ID does not exist"
Oct 31 07:57:57.297211 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.297192    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-runtime-predictor-74d5699c88-9bnk5"]
Oct 31 07:57:57.704013 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:57.703940    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" path="/var/lib/kubelet/pods/50dfd486-658e-4c6a-9e8e-373868947e94/volumes"
Oct 31 07:57:58.832998 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:58.832960    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:58.940042 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:57:58.939991    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:59.788151 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:57:59.788118    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:58:02.405004 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:58:02.404968    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:08.698631 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:58:08.698593    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:13.192306 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:58:13.192277    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:13.379345 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:58:13.379313    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:29.788919 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:58:29.788882    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:58:59.790428 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:58:59.790383    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:59:02.456788 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:59:02.456753    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:29.789487 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:59:29.789448    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 07:59:40.023236 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:59:40.023208    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:47.427091 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:59:47.427053    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:47.528122 ip-10-0-137-69 kubenswrapper[4371]: I1031 07:59:47.528088    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:59.789597 ip-10-0-137-69 kubenswrapper[4371]: E1031 07:59:59.789562    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:00:02.492320 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:00:02.492232    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:09.695954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:00:09.695921    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:12.084789 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:00:12.084757    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:12.289679 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:00:12.289648    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:29.789635 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:00:29.789599    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:00:59.788660 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:00:59.788622    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:01:02.535468 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:02.535429    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:18.509260 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:18.509224    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:25.925972 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.925902    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926273    4371 cpu_manager.go:414] "containerMap: removing container" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kube-rbac-proxy"
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926288    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kube-rbac-proxy"
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926310    4371 cpu_manager.go:414] "containerMap: removing container" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="storage-initializer"
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926316    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="storage-initializer"
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926324    4371 cpu_manager.go:414] "containerMap: removing container" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kserve-container"
Oct 31 08:01:25.926343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926329    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kserve-container"
Oct 31 08:01:25.926576 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926383    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kube-rbac-proxy"
Oct 31 08:01:25.926576 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.926393    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dfd486-658e-4c6a-9e8e-373868947e94" containerName="kserve-container"
Oct 31 08:01:25.929490 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.929475    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:25.934806 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.934783    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:01:25.940222 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.940201    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:01:25.940355 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.940212    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Oct 31 08:01:25.940414 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.940228    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:01:25.940414 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.940255    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:01:25.941805 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.941781    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:01:25.945096 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:25.945076    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:26.011361 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.011336    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.011541 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.011381    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqst\" (UniqueName: \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.011541 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.011407    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.011651 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.011562    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.011651 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.011588    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.014344 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.014324    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:01:26.014465 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.014452    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-tensorflow-predictor-serving-cert\""
Oct 31 08:01:26.098663 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.098635    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:26.112750 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.112729    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.112832 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.112769    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqst\" (UniqueName: \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.112832 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.112797    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.112832 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.112829    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.112985 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.112846    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.113230 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.113211    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.113431 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.113415    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.113474 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.113453    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.115315 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.115291    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.121965 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.121939    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqst\" (UniqueName: \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\") pod \"isvc-tensorflow-predictor-6d644d4b5b-sgf4z\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.253574 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.253537    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:26.524903 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:26.524880    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:01:26.526306 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:01:26.526280    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a359d2_eb4d_448a_94d1_f16ad62a794c.slice/crio-7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a WatchSource:0}: Error finding container 7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a: Status 404 returned error can't find the container with id 7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a
Oct 31 08:01:27.019155 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:27.019124    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerStarted","Data":"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"}
Oct 31 08:01:27.019548 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:27.019176    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerStarted","Data":"7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a"}
Oct 31 08:01:29.790216 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:01:29.790180    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:01:29.817582 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:29.817543    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:32.038135 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:32.038052    4371 generic.go:356] "Generic (PLEG): container finished" podID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerID="9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190" exitCode=0
Oct 31 08:01:32.038135 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:32.038114    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerDied","Data":"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"}
Oct 31 08:01:32.039495 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:32.039480    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:01:36.057612 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.057503    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerStarted","Data":"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"}
Oct 31 08:01:36.057612 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.057601    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerStarted","Data":"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"}
Oct 31 08:01:36.058050 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.057878    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:36.058099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.058049    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:36.059342 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.059318    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 31 08:01:36.079105 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:36.079063    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podStartSLOduration=2.185723621 podStartE2EDuration="11.079051808s" totalImagesPullingTime="3.748391523s" totalInitContainerRuntime="5.144936664s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:01:36.07737057 +0000 UTC m=+1938.493501266" watchObservedRunningTime="2026-10-31 08:01:36.079051808 +0000 UTC m=+1938.495182471"
Oct 31 08:01:37.062051 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:37.061996    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 31 08:01:42.066278 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:42.066251    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:42.066781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:42.066700    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 31 08:01:52.067580 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:52.067552    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:01:59.165187 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:01:59.165152    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:59.789264 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:01:59.789223    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:02:01.650020 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:01.649986    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:01.755216 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:01.755174    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:02.570099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:02.570063    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:11.164294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:11.164259    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:02:11.164795 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:11.164580    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container" containerID="cri-o://b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8" gracePeriod=30
Oct 31 08:02:11.164795 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:11.164656    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" containerID="cri-o://95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228" gracePeriod=30
Oct 31 08:02:11.221704 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:11.221675    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:12.062531 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:12.062465    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:12.189899 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:12.189866    4371 generic.go:356] "Generic (PLEG): container finished" podID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerID="95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228" exitCode=0
Oct 31 08:02:12.190295 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:12.189941    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerDied","Data":"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"}
Oct 31 08:02:17.063140 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:17.063102    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:22.062679 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:22.062636    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:22.063086 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:22.062775    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:02:27.062394 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:27.062347    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:29.789951 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:29.789911    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:02:32.062923 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:32.062877    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:37.062832 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:37.062786    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 31 08:02:41.196084 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:41.196038    4371 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a359d2_eb4d_448a_94d1_f16ad62a794c.slice/crio-7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a\": RecentStats: unable to find data in memory cache]"
Oct 31 08:02:41.807840 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.807812    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:02:41.899468 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899437    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\") pod \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") "
Oct 31 08:02:41.899468 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899471    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\") pod \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") "
Oct 31 08:02:41.899692 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899532    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\") pod \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") "
Oct 31 08:02:41.899692 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899598    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\" (UniqueName: \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\") pod \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") "
Oct 31 08:02:41.899692 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899627    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\") pod \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\" (UID: \"c8a359d2-eb4d-448a-94d1-f16ad62a794c\") "
Oct 31 08:02:41.899888 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899860    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert" pod "c8a359d2-eb4d-448a-94d1-f16ad62a794c" (UID: "c8a359d2-eb4d-448a-94d1-f16ad62a794c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:02:41.899954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.899891    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config" pod "c8a359d2-eb4d-448a-94d1-f16ad62a794c" (UID: "c8a359d2-eb4d-448a-94d1-f16ad62a794c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:02:41.901693 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.901667    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls" pod "c8a359d2-eb4d-448a-94d1-f16ad62a794c" (UID: "c8a359d2-eb4d-448a-94d1-f16ad62a794c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:02:41.901876 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.901856    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst" pod "c8a359d2-eb4d-448a-94d1-f16ad62a794c" (UID: "c8a359d2-eb4d-448a-94d1-f16ad62a794c"). InnerVolumeSpecName "kube-api-access-kwqst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:02:41.910054 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:41.910031    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location" pod "c8a359d2-eb4d-448a-94d1-f16ad62a794c" (UID: "c8a359d2-eb4d-448a-94d1-f16ad62a794c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:02:42.000989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.000957    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a359d2-eb4d-448a-94d1-f16ad62a794c-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:42.000989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.000988    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:42.001234 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.000999    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:42.001234 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.001009    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:42.001234 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.001018    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwqst\" (UniqueName: \"kubernetes.io/projected/c8a359d2-eb4d-448a-94d1-f16ad62a794c-kube-api-access-kwqst\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:42.303289 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.303195    4371 generic.go:356] "Generic (PLEG): container finished" podID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerID="b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8" exitCode=137
Oct 31 08:02:42.303289 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.303249    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"
Oct 31 08:02:42.303767 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.303250    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerDied","Data":"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"}
Oct 31 08:02:42.303767 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.303382    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z" event={"ID":"c8a359d2-eb4d-448a-94d1-f16ad62a794c","Type":"ContainerDied","Data":"7abdb7934902e105e106ff2cd2b182a64fe3bec8691e9ef5e20bf3003b00762a"}
Oct 31 08:02:42.303767 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.303419    4371 scope.go:122] "RemoveContainer" containerID="95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"
Oct 31 08:02:42.314324 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.314190    4371 scope.go:122] "RemoveContainer" containerID="b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"
Oct 31 08:02:42.322192 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.322173    4371 scope.go:122] "RemoveContainer" containerID="9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"
Oct 31 08:02:42.328877 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.328855    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:02:42.329585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.329570    4371 scope.go:122] "RemoveContainer" containerID="95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"
Oct 31 08:02:42.329864 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:42.329841    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228\": container with ID starting with 95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228 not found: ID does not exist" containerID="95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"
Oct 31 08:02:42.329955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.329876    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228"} err="failed to get container status \"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228\": rpc error: code = NotFound desc = could not find container \"95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228\": container with ID starting with 95e8f0430474f931768690721992a0ded1238c2a7819799283a020178cd5c228 not found: ID does not exist"
Oct 31 08:02:42.329955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.329900    4371 scope.go:122] "RemoveContainer" containerID="b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"
Oct 31 08:02:42.330227 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:42.330202    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8\": container with ID starting with b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8 not found: ID does not exist" containerID="b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"
Oct 31 08:02:42.330382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.330263    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8"} err="failed to get container status \"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8\": rpc error: code = NotFound desc = could not find container \"b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8\": container with ID starting with b9deefe701e10505e8dfc900b56e16e4bf5a1ea05173270272257eed538aecb8 not found: ID does not exist"
Oct 31 08:02:42.330382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.330285    4371 scope.go:122] "RemoveContainer" containerID="9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"
Oct 31 08:02:42.330619 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:42.330590    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190\": container with ID starting with 9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190 not found: ID does not exist" containerID="9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"
Oct 31 08:02:42.330727 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.330624    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190"} err="failed to get container status \"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190\": rpc error: code = NotFound desc = could not find container \"9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190\": container with ID starting with 9f89d384795387226c3f3b9e0acfe05b331bf16e45a357936b555e079f260190 not found: ID does not exist"
Oct 31 08:02:42.332070 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.332052    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-predictor-6d644d4b5b-sgf4z"]
Oct 31 08:02:42.978087 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978052    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:02:42.978460 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978443    4371 cpu_manager.go:414] "containerMap: removing container" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="storage-initializer"
Oct 31 08:02:42.978577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978463    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="storage-initializer"
Oct 31 08:02:42.978577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978497    4371 cpu_manager.go:414] "containerMap: removing container" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy"
Oct 31 08:02:42.978577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978520    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy"
Oct 31 08:02:42.978577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978536    4371 cpu_manager.go:414] "containerMap: removing container" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container"
Oct 31 08:02:42.978577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978543    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container"
Oct 31 08:02:42.978819 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978641    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kube-rbac-proxy"
Oct 31 08:02:42.978819 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.978654    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" containerName="kserve-container"
Oct 31 08:02:42.983255 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.983233    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:42.990599 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.990577    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:02:42.990722 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.990666    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:02:42.990798 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.990778    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:02:42.991351 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.991328    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:02:42.991463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.991386    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Oct 31 08:02:42.991463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.991388    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:02:42.991463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.991405    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:02:42.991690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.991672    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:42.993204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:42.993183    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:02:43.099155 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.099121    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:43.110181 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.110150    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.110316 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.110191    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.110357 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.110318    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxmh\" (UniqueName: \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.110357 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.110348    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.110435 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.110367    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211124 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211083    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxmh\" (UniqueName: \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211124 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211131    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211149    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211168    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211202    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211592 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211565    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.211989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211957    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.212098 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.211957    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.213758 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.213729    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.219655 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.219628    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxmh\" (UniqueName: \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\") pod \"isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") " pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.301484 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.301396    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:43.573041 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.573016    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:02:43.575216 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:02:43.575190    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac077cb5_ffa1_4d38_9faf_a08e23943be1.slice/crio-6213e25ba324845a8946666475ae8ece60a5e6b00e0336ec6c46a349bbc97702 WatchSource:0}: Error finding container 6213e25ba324845a8946666475ae8ece60a5e6b00e0336ec6c46a349bbc97702: Status 404 returned error can't find the container with id 6213e25ba324845a8946666475ae8ece60a5e6b00e0336ec6c46a349bbc97702
Oct 31 08:02:43.703486 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:43.703439    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a359d2-eb4d-448a-94d1-f16ad62a794c" path="/var/lib/kubelet/pods/c8a359d2-eb4d-448a-94d1-f16ad62a794c/volumes"
Oct 31 08:02:44.313536 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:44.313468    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerStarted","Data":"6c9967013b13db36360bcebbc087155b34aea7b3163ec1093778fb3695507536"}
Oct 31 08:02:44.313718 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:44.313685    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerStarted","Data":"6213e25ba324845a8946666475ae8ece60a5e6b00e0336ec6c46a349bbc97702"}
Oct 31 08:02:49.333760 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:49.333726    4371 generic.go:356] "Generic (PLEG): container finished" podID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerID="6c9967013b13db36360bcebbc087155b34aea7b3163ec1093778fb3695507536" exitCode=0
Oct 31 08:02:49.333760 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:49.333765    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerDied","Data":"6c9967013b13db36360bcebbc087155b34aea7b3163ec1093778fb3695507536"}
Oct 31 08:02:50.339945 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.339912    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerStarted","Data":"e2c617213daa7a983d5cdbc35bdffa2dd9ea4115413ba635dd3ae8a4b7bb0a9a"}
Oct 31 08:02:50.340435 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.339980    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerStarted","Data":"4d3acd5d7683cd5cf7e593c7e3e3aac975f5c3a605232be0b060a48698cae038"}
Oct 31 08:02:50.340435 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.340256    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:50.340435 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.340301    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:50.341592 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.341566    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 31 08:02:50.368021 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:50.367975    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podStartSLOduration=3.374363727 podStartE2EDuration="8.367962944s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.993599217s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:50.366495335 +0000 UTC m=+2012.782626008" watchObservedRunningTime="2026-10-31 08:02:50.367962944 +0000 UTC m=+2012.784093660"
Oct 31 08:02:51.343544 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:51.343485    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 31 08:02:56.348061 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:56.348026    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:02:56.348688 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:02:56.348648    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 31 08:02:59.788593 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:02:59.788558    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:03:02.625035 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:02.624964    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:06.349058 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:06.349031    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:03:23.270011 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.269976    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:03:23.270497 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.270314    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container" containerID="cri-o://4d3acd5d7683cd5cf7e593c7e3e3aac975f5c3a605232be0b060a48698cae038" gracePeriod=30
Oct 31 08:03:23.270497 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.270341    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" containerID="cri-o://e2c617213daa7a983d5cdbc35bdffa2dd9ea4115413ba635dd3ae8a4b7bb0a9a" gracePeriod=30
Oct 31 08:03:23.334536 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.334500    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:23.464795 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.464766    4371 generic.go:356] "Generic (PLEG): container finished" podID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerID="e2c617213daa7a983d5cdbc35bdffa2dd9ea4115413ba635dd3ae8a4b7bb0a9a" exitCode=0
Oct 31 08:03:23.464911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:23.464836    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerDied","Data":"e2c617213daa7a983d5cdbc35bdffa2dd9ea4115413ba635dd3ae8a4b7bb0a9a"}
Oct 31 08:03:26.343857 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:26.343815    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:29.789533 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:03:29.789475    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:03:31.343683 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:31.343651    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:36.344132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:36.344082    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:36.344639 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:36.344228    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:03:41.343926 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:41.343845    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:46.344642 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:46.344598    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:51.344527 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:51.344471    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 31 08:03:53.574784 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.574697    4371 generic.go:356] "Generic (PLEG): container finished" podID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerID="4d3acd5d7683cd5cf7e593c7e3e3aac975f5c3a605232be0b060a48698cae038" exitCode=137
Oct 31 08:03:53.574784 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.574746    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerDied","Data":"4d3acd5d7683cd5cf7e593c7e3e3aac975f5c3a605232be0b060a48698cae038"}
Oct 31 08:03:53.911959 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.911927    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:03:53.965845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.965818    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\") pod \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") "
Oct 31 08:03:53.966006 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.965895    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\") pod \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") "
Oct 31 08:03:53.966006 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.965920    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\") pod \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") "
Oct 31 08:03:53.966006 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.965962    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\") pod \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") "
Oct 31 08:03:53.966172 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.966014    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\" (UniqueName: \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\") pod \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\" (UID: \"ac077cb5-ffa1-4d38-9faf-a08e23943be1\") "
Oct 31 08:03:53.966227 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.966193    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config" pod "ac077cb5-ffa1-4d38-9faf-a08e23943be1" (UID: "ac077cb5-ffa1-4d38-9faf-a08e23943be1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:53.966380 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.966357    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert" pod "ac077cb5-ffa1-4d38-9faf-a08e23943be1" (UID: "ac077cb5-ffa1-4d38-9faf-a08e23943be1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:53.968004 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.967978    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls" pod "ac077cb5-ffa1-4d38-9faf-a08e23943be1" (UID: "ac077cb5-ffa1-4d38-9faf-a08e23943be1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:03:53.968004 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.967982    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh" pod "ac077cb5-ffa1-4d38-9faf-a08e23943be1" (UID: "ac077cb5-ffa1-4d38-9faf-a08e23943be1"). InnerVolumeSpecName "kube-api-access-tdxmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:53.977281 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:53.977258    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location" pod "ac077cb5-ffa1-4d38-9faf-a08e23943be1" (UID: "ac077cb5-ffa1-4d38-9faf-a08e23943be1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:54.067226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.067181    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxmh\" (UniqueName: \"kubernetes.io/projected/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-api-access-tdxmh\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:54.067226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.067222    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:54.067226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.067232    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac077cb5-ffa1-4d38-9faf-a08e23943be1-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:54.067226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.067243    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac077cb5-ffa1-4d38-9faf-a08e23943be1-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:54.067458 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.067253    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac077cb5-ffa1-4d38-9faf-a08e23943be1-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:54.580440 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.580413    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"
Oct 31 08:03:54.580885 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.580403    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr" event={"ID":"ac077cb5-ffa1-4d38-9faf-a08e23943be1","Type":"ContainerDied","Data":"6213e25ba324845a8946666475ae8ece60a5e6b00e0336ec6c46a349bbc97702"}
Oct 31 08:03:54.580885 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.580572    4371 scope.go:122] "RemoveContainer" containerID="e2c617213daa7a983d5cdbc35bdffa2dd9ea4115413ba635dd3ae8a4b7bb0a9a"
Oct 31 08:03:54.589009 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.588993    4371 scope.go:122] "RemoveContainer" containerID="4d3acd5d7683cd5cf7e593c7e3e3aac975f5c3a605232be0b060a48698cae038"
Oct 31 08:03:54.596169 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.596152    4371 scope.go:122] "RemoveContainer" containerID="6c9967013b13db36360bcebbc087155b34aea7b3163ec1093778fb3695507536"
Oct 31 08:03:54.603761 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.603736    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:03:54.605802 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:54.605781    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-tensorflow-runtime-predictor-8569dc6ddd-zmsbr"]
Oct 31 08:03:55.211197 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211163    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:03:55.211480 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211468    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy"
Oct 31 08:03:55.211547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211481    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy"
Oct 31 08:03:55.211547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211499    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="storage-initializer"
Oct 31 08:03:55.211547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211504    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="storage-initializer"
Oct 31 08:03:55.211547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211524    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container"
Oct 31 08:03:55.211547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211530    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container"
Oct 31 08:03:55.211701 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211579    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kube-rbac-proxy"
Oct 31 08:03:55.211701 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.211588    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" containerName="kserve-container"
Oct 31 08:03:55.213962 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.213899    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:55.216153 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.216139    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.224797 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224774    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:03:55.224914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224803    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:03:55.224914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224806    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:03:55.224914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224775    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Oct 31 08:03:55.224914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224837    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:03:55.224914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224861    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-triton-predictor-serving-cert\""
Oct 31 08:03:55.225172 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.224920    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:03:55.226445 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.226419    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:03:55.276060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.276033    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.276219 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.276070    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.276219 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.276101    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.276311 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.276221    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkw9w\" (UniqueName: \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.276311 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.276251    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.310176 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.310150    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:55.376812 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.376780    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.376933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.376816    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.376933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.376846    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.376933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.376872    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkw9w\" (UniqueName: \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.376933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.376902    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.377340 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.377316    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.377570 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.377551    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.377620 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.377565    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.379419 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.379403    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.384481 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.384457    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkw9w\" (UniqueName: \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\") pod \"isvc-triton-predictor-79f59d6cd9-qldv9\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") " pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.534404 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.534369    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:03:55.703251 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.703218    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac077cb5-ffa1-4d38-9faf-a08e23943be1" path="/var/lib/kubelet/pods/ac077cb5-ffa1-4d38-9faf-a08e23943be1/volumes"
Oct 31 08:03:55.808158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:55.808133    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:03:55.810081 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:03:55.810048    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fbb8bd_a52d_424d_8474_88584f170cdc.slice/crio-cc91192d884fbfcdd8121cb3a3d0c3f63a0200344150236bfd96b8d50c06e3e7 WatchSource:0}: Error finding container cc91192d884fbfcdd8121cb3a3d0c3f63a0200344150236bfd96b8d50c06e3e7: Status 404 returned error can't find the container with id cc91192d884fbfcdd8121cb3a3d0c3f63a0200344150236bfd96b8d50c06e3e7
Oct 31 08:03:56.593158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:56.593121    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerStarted","Data":"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"}
Oct 31 08:03:56.593341 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:56.593190    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerStarted","Data":"cc91192d884fbfcdd8121cb3a3d0c3f63a0200344150236bfd96b8d50c06e3e7"}
Oct 31 08:03:58.089533 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:03:58.089484    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:59.789648 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:03:59.789610    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:04:00.479341 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:00.479312    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:00.586655 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:00.586623    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:00.608838 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:00.608806    4371 generic.go:356] "Generic (PLEG): container finished" podID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerID="9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85" exitCode=0
Oct 31 08:04:00.608999 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:00.608879    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerDied","Data":"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"}
Oct 31 08:04:02.667129 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:02.667081    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:03.153080 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:03.153044    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:29.794094 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:04:29.794056    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:04:43.313471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:43.313434    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:47.737629 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:47.737587    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:47.839723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:04:47.839682    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:59.927359 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:04:59.927319    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:05:02.718360 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:02.718329    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:29.838030 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:05:29.837986    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:05:33.450152 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:33.450114    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:36.776844 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.776804    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:36.893043 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.893003    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:36.899297 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.899263    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:05:36.921203 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.921170    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:05:36.921390 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.921315    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:36.925535 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.925353    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 31 08:05:36.925535 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.925401    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:05:36.925535 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.925436    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:05:36.925535 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.925488    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:05:36.926967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.926927    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 31 08:05:36.926967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.926943    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:05:36.927142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.927019    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:05:36.996478 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:36.996434    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:37.004631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.004595    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.004809 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.004709    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.004809 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.004754    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdtw\" (UniqueName: \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.004809 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.004780    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.004974 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.004948    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.105917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.105820    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.105917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.105885    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.105917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.105908    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdtw\" (UniqueName: \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.105917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.105923    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.106240 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.105983    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.106396 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.106363    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.106654 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.106634    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.106748 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.106706    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.108729 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.108706    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.114444 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.114417    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdtw\" (UniqueName: \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\") pod \"isvc-canary-create-predictor-55d965bd48-2qcd2\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.281119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.280848    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:05:37.615836 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:37.615811    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:05:38.081262 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:38.081217    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerStarted","Data":"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"}
Oct 31 08:05:38.081730 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:38.081285    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerStarted","Data":"d2c22665afb19872a9097068102db6fa6d4ee1b013a9815b35819ff80174db29"}
Oct 31 08:05:43.110538 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:43.110274    4371 generic.go:356] "Generic (PLEG): container finished" podID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerID="cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6" exitCode=0
Oct 31 08:05:43.110538 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:43.110416    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerDied","Data":"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"}
Oct 31 08:05:58.514669 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:05:58.514632    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:59.792477 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:05:59.792442    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:06:02.746354 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:02.746275    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:09.237471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.237436    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerStarted","Data":"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"}
Oct 31 08:06:09.237965 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.237524    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerStarted","Data":"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"}
Oct 31 08:06:09.237965 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.237751    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:06:09.239768 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.239744    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerStarted","Data":"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"}
Oct 31 08:06:09.239892 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.239789    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerStarted","Data":"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"}
Oct 31 08:06:09.240037 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.240015    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:09.240106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.240070    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:09.241501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.241477    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 31 08:06:09.258370 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.258317    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podStartSLOduration=2.333743716 podStartE2EDuration="33.258304833s" totalImagesPullingTime="25.977356421s" totalInitContainerRuntime="4.947204696s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:06:09.256262347 +0000 UTC m=+2211.672393018" watchObservedRunningTime="2026-10-31 08:06:09.258304833 +0000 UTC m=+2211.674435505"
Oct 31 08:06:09.274720 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:09.274669    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podStartSLOduration=2.5309877 podStartE2EDuration="2m14.274655812s" totalImagesPullingTime="2m7.882331471s" totalInitContainerRuntime="3.861336641s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:06:09.273424882 +0000 UTC m=+2211.689555567" watchObservedRunningTime="2026-10-31 08:06:09.274655812 +0000 UTC m=+2211.690786551"
Oct 31 08:06:10.244319 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:10.244269    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 31 08:06:10.244319 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:10.244308    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:06:10.245428 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:10.245401    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:11.247868 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:11.247829    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:15.249279 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:15.249248    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:15.249866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:15.249841    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 31 08:06:16.252616 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:16.252588    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:06:16.253114 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:16.253041    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:24.086095 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:24.086057    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:25.250900 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:25.250862    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:26.253821 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:26.253786    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:29.790280 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:06:29.790241    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:06:36.253135 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:36.253095    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:37.669089 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:37.669052    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:06:37.669540 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:37.669349    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container" containerID="cri-o://e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e" gracePeriod=30
Oct 31 08:06:37.669540 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:37.669377    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kube-rbac-proxy" containerID="cri-o://55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a" gracePeriod=30
Oct 31 08:06:37.738674 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:37.738621    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:38.354182 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:38.354090    4371 generic.go:356] "Generic (PLEG): container finished" podID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerID="55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a" exitCode=0
Oct 31 08:06:38.354182 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:38.354157    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerDied","Data":"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"}
Oct 31 08:06:40.245114 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.245075    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 31 08:06:40.652588 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.652554    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:40.753813 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.753730    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\") pod \"35fbb8bd-a52d-424d-8474-88584f170cdc\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") "
Oct 31 08:06:40.753813 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.753774    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\") pod \"35fbb8bd-a52d-424d-8474-88584f170cdc\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") "
Oct 31 08:06:40.754011 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.753874    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\") pod \"35fbb8bd-a52d-424d-8474-88584f170cdc\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") "
Oct 31 08:06:40.754011 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.753928    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\" (UniqueName: \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\") pod \"35fbb8bd-a52d-424d-8474-88584f170cdc\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") "
Oct 31 08:06:40.754011 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.753980    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\") pod \"35fbb8bd-a52d-424d-8474-88584f170cdc\" (UID: \"35fbb8bd-a52d-424d-8474-88584f170cdc\") "
Oct 31 08:06:40.754148 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.754120    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config" pod "35fbb8bd-a52d-424d-8474-88584f170cdc" (UID: "35fbb8bd-a52d-424d-8474-88584f170cdc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:06:40.754292 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.754166    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location" pod "35fbb8bd-a52d-424d-8474-88584f170cdc" (UID: "35fbb8bd-a52d-424d-8474-88584f170cdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:40.754413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.754386    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fbb8bd-a52d-424d-8474-88584f170cdc-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:06:40.754496 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.754419    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:06:40.754496 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.754393    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert" pod "35fbb8bd-a52d-424d-8474-88584f170cdc" (UID: "35fbb8bd-a52d-424d-8474-88584f170cdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:06:40.755996 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.755978    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls" pod "35fbb8bd-a52d-424d-8474-88584f170cdc" (UID: "35fbb8bd-a52d-424d-8474-88584f170cdc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:06:40.756090 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.756021    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w" pod "35fbb8bd-a52d-424d-8474-88584f170cdc" (UID: "35fbb8bd-a52d-424d-8474-88584f170cdc"). InnerVolumeSpecName "kube-api-access-nkw9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:06:40.855217 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.855177    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkw9w\" (UniqueName: \"kubernetes.io/projected/35fbb8bd-a52d-424d-8474-88584f170cdc-kube-api-access-nkw9w\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:06:40.855217 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.855212    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fbb8bd-a52d-424d-8474-88584f170cdc-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:06:40.855217 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:40.855222    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fbb8bd-a52d-424d-8474-88584f170cdc-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.368954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.368872    4371 generic.go:356] "Generic (PLEG): container finished" podID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerID="e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e" exitCode=0
Oct 31 08:06:41.368954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.368918    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerDied","Data":"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"}
Oct 31 08:06:41.368954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.368932    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"
Oct 31 08:06:41.369466 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.368963    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9" event={"ID":"35fbb8bd-a52d-424d-8474-88584f170cdc","Type":"ContainerDied","Data":"cc91192d884fbfcdd8121cb3a3d0c3f63a0200344150236bfd96b8d50c06e3e7"}
Oct 31 08:06:41.369466 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.368988    4371 scope.go:122] "RemoveContainer" containerID="55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"
Oct 31 08:06:41.377751 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.377732    4371 scope.go:122] "RemoveContainer" containerID="e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"
Oct 31 08:06:41.385240 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.385213    4371 scope.go:122] "RemoveContainer" containerID="9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"
Oct 31 08:06:41.391501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.391474    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:06:41.392447 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.392432    4371 scope.go:122] "RemoveContainer" containerID="55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"
Oct 31 08:06:41.392728 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:06:41.392707    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a\": container with ID starting with 55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a not found: ID does not exist" containerID="55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"
Oct 31 08:06:41.392794 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.392737    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a"} err="failed to get container status \"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a\": rpc error: code = NotFound desc = could not find container \"55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a\": container with ID starting with 55f83690fd77342b7bdd8bc87115ce7bb739e93029602a6d33eabd862d9d8a4a not found: ID does not exist"
Oct 31 08:06:41.392794 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.392755    4371 scope.go:122] "RemoveContainer" containerID="e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"
Oct 31 08:06:41.392965 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:06:41.392952    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e\": container with ID starting with e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e not found: ID does not exist" containerID="e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"
Oct 31 08:06:41.393005 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.392969    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e"} err="failed to get container status \"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e\": rpc error: code = NotFound desc = could not find container \"e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e\": container with ID starting with e8ef0b83a0a0c3d5c2fe1cb436b3cd766ea6aed0fc3a66f83374b732751efa5e not found: ID does not exist"
Oct 31 08:06:41.393005 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.392979    4371 scope.go:122] "RemoveContainer" containerID="9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"
Oct 31 08:06:41.393255 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:06:41.393231    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85\": container with ID starting with 9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85 not found: ID does not exist" containerID="9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"
Oct 31 08:06:41.393362 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.393281    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85"} err="failed to get container status \"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85\": rpc error: code = NotFound desc = could not find container \"9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85\": container with ID starting with 9e113daa20ab27a20beb5e33cc8dc18469b57092c4a00c999c909c7e4636ae85 not found: ID does not exist"
Oct 31 08:06:41.395172 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.395149    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-triton-predictor-79f59d6cd9-qldv9"]
Oct 31 08:06:41.704283 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:41.704206    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" path="/var/lib/kubelet/pods/35fbb8bd-a52d-424d-8474-88584f170cdc/volumes"
Oct 31 08:06:43.098619 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.098588    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:06:43.099025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.098955    4371 cpu_manager.go:414] "containerMap: removing container" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kube-rbac-proxy"
Oct 31 08:06:43.099025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.098971    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kube-rbac-proxy"
Oct 31 08:06:43.099025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099003    4371 cpu_manager.go:414] "containerMap: removing container" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container"
Oct 31 08:06:43.099025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099013    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container"
Oct 31 08:06:43.099025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099026    4371 cpu_manager.go:414] "containerMap: removing container" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="storage-initializer"
Oct 31 08:06:43.099236 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099032    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="storage-initializer"
Oct 31 08:06:43.099236 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099086    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kserve-container"
Oct 31 08:06:43.099236 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099099    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fbb8bd-a52d-424d-8474-88584f170cdc" containerName="kube-rbac-proxy"
Oct 31 08:06:43.099645 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.099623    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:43.132115 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.132083    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:06:43.132286 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.132226    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.136423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.136400    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:06:43.136589 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.136400    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Oct 31 08:06:43.136673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.136425    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:06:43.137866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.137847    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:06:43.137866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.137853    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:06:43.138010 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.137854    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-predictor-serving-cert\""
Oct 31 08:06:43.138010 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.137891    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:06:43.197140 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.197102    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:43.276519 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.276474    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.276687 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.276541    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.276687 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.276631    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5jq\" (UniqueName: \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.276687 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.276673    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.276816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.276748    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377081    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377116    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5jq\" (UniqueName: \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377137    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377372 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377264    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377372 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377364    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377764 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377739    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.377958 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377929    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.378042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.377927    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.379639 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.379615    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.385326 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.385300    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5jq\" (UniqueName: \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\") pod \"isvc-xgboost-predictor-75fd7f49d4-mcxf5\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.505151 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.505112    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:06:43.805942 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:43.805915    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:06:43.808438 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:06:43.808406    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186deef1_25e7_4595_9932_31720a5f5c90.slice/crio-6206a61616f0c84b57921d5a4f5e82dc0e5fbf982d4cd8bf5688620286356962 WatchSource:0}: Error finding container 6206a61616f0c84b57921d5a4f5e82dc0e5fbf982d4cd8bf5688620286356962: Status 404 returned error can't find the container with id 6206a61616f0c84b57921d5a4f5e82dc0e5fbf982d4cd8bf5688620286356962
Oct 31 08:06:44.384333 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:44.384291    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerStarted","Data":"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"}
Oct 31 08:06:44.384850 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:44.384375    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerStarted","Data":"6206a61616f0c84b57921d5a4f5e82dc0e5fbf982d4cd8bf5688620286356962"}
Oct 31 08:06:46.253979 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:46.253932    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:49.408296 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:49.408257    4371 generic.go:356] "Generic (PLEG): container finished" podID="186deef1-25e7-4595-9932-31720a5f5c90" containerID="d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284" exitCode=0
Oct 31 08:06:49.408744 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:49.408330    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerDied","Data":"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"}
Oct 31 08:06:49.409684 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:49.409657    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:06:56.253372 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:06:56.253334    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:06:59.792500 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:06:59.792460    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:07:02.794588 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:02.794552    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:06.253392 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:06.253347    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:07:08.500747 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.500713    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerStarted","Data":"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"}
Oct 31 08:07:08.501202 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.500775    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerStarted","Data":"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"}
Oct 31 08:07:08.501202 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.501101    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:07:08.501294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.501235    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:07:08.502610 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.502585    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:08.521815 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:08.521767    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podStartSLOduration=2.414172126 podStartE2EDuration="25.521756433s" totalImagesPullingTime="18.57157748s" totalInitContainerRuntime="4.536006827s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:08.519874577 +0000 UTC m=+2270.936005249" watchObservedRunningTime="2026-10-31 08:07:08.521756433 +0000 UTC m=+2270.937887106"
Oct 31 08:07:09.504504 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:09.504461    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:14.508664 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:14.508634    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:07:14.509112 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:14.509084    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:16.253315 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:16.253275    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:07:22.699167 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:22.699117    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 31 08:07:24.509156 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:24.509116    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:29.791013 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:07:29.790973    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:07:32.699747 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:32.699667    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:07:34.509294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:34.509247    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:37.100848 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.100799    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:07:37.101335 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.101122    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container" containerID="cri-o://a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a" gracePeriod=30
Oct 31 08:07:37.102216 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.101630    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kube-rbac-proxy" containerID="cri-o://0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746" gracePeriod=30
Oct 31 08:07:37.169022 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.168990    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:37.264065 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.264035    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:37.617055 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.617027    4371 generic.go:356] "Generic (PLEG): container finished" podID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerID="0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746" exitCode=0
Oct 31 08:07:37.617219 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:37.617087    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerDied","Data":"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"}
Oct 31 08:07:41.248881 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:41.248841    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Oct 31 08:07:42.222809 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.222780    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:07:42.359107 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359075    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\") pod \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") "
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359135    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\") pod \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") "
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359156    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\") pod \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") "
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359194    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\") pod \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") "
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359223    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\" (UniqueName: \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\") pod \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\" (UID: \"34f7dfe2-0b4f-4f47-a33e-0db808e8b362\") "
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359505    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location" pod "34f7dfe2-0b4f-4f47-a33e-0db808e8b362" (UID: "34f7dfe2-0b4f-4f47-a33e-0db808e8b362"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359568    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert" pod "34f7dfe2-0b4f-4f47-a33e-0db808e8b362" (UID: "34f7dfe2-0b4f-4f47-a33e-0db808e8b362"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:42.359600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.359563    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config" pod "34f7dfe2-0b4f-4f47-a33e-0db808e8b362" (UID: "34f7dfe2-0b4f-4f47-a33e-0db808e8b362"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:42.361439 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.361405    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls" pod "34f7dfe2-0b4f-4f47-a33e-0db808e8b362" (UID: "34f7dfe2-0b4f-4f47-a33e-0db808e8b362"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:42.361597 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.361546    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw" pod "34f7dfe2-0b4f-4f47-a33e-0db808e8b362" (UID: "34f7dfe2-0b4f-4f47-a33e-0db808e8b362"). InnerVolumeSpecName "kube-api-access-ptdtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:42.460783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.460692    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.460783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.460732    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdtw\" (UniqueName: \"kubernetes.io/projected/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-api-access-ptdtw\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.460783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.460742    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.460783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.460751    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.460783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.460760    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f7dfe2-0b4f-4f47-a33e-0db808e8b362-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.640467 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.640426    4371 generic.go:356] "Generic (PLEG): container finished" podID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerID="a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a" exitCode=0
Oct 31 08:07:42.640631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.640484    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerDied","Data":"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"}
Oct 31 08:07:42.640631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.640497    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"
Oct 31 08:07:42.640631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.640584    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2" event={"ID":"34f7dfe2-0b4f-4f47-a33e-0db808e8b362","Type":"ContainerDied","Data":"d2c22665afb19872a9097068102db6fa6d4ee1b013a9815b35819ff80174db29"}
Oct 31 08:07:42.640631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.640613    4371 scope.go:122] "RemoveContainer" containerID="0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"
Oct 31 08:07:42.648974 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.648955    4371 scope.go:122] "RemoveContainer" containerID="a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"
Oct 31 08:07:42.656400 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.656384    4371 scope.go:122] "RemoveContainer" containerID="cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"
Oct 31 08:07:42.663547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.663529    4371 scope.go:122] "RemoveContainer" containerID="0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"
Oct 31 08:07:42.663797 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:07:42.663780    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746\": container with ID starting with 0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746 not found: ID does not exist" containerID="0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"
Oct 31 08:07:42.663852 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.663806    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746"} err="failed to get container status \"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746\": rpc error: code = NotFound desc = could not find container \"0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746\": container with ID starting with 0627c072d0bf40a19a8b602ff74532b57f715fe0de1f7efc580ace98ea561746 not found: ID does not exist"
Oct 31 08:07:42.663852 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.663824    4371 scope.go:122] "RemoveContainer" containerID="a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"
Oct 31 08:07:42.664013 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.663994    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:07:42.664055 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:07:42.664026    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a\": container with ID starting with a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a not found: ID does not exist" containerID="a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"
Oct 31 08:07:42.664055 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.664041    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a"} err="failed to get container status \"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a\": rpc error: code = NotFound desc = could not find container \"a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a\": container with ID starting with a9aa0a3dabe959bb852a5807b3b51761642c6b5b846d6aa55d7f5552b60d586a not found: ID does not exist"
Oct 31 08:07:42.664055 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.664052    4371 scope.go:122] "RemoveContainer" containerID="cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"
Oct 31 08:07:42.664259 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:07:42.664241    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6\": container with ID starting with cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6 not found: ID does not exist" containerID="cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"
Oct 31 08:07:42.664300 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.664266    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6"} err="failed to get container status \"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6\": rpc error: code = NotFound desc = could not find container \"cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6\": container with ID starting with cad57adb71a602b5c07975963bc5e81d80b29b87d35c020de3d4cbe4c35b89d6 not found: ID does not exist"
Oct 31 08:07:42.669745 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:42.669726    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-2qcd2"]
Oct 31 08:07:43.665570 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.665533    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:43.703485 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.703451    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" path="/var/lib/kubelet/pods/34f7dfe2-0b4f-4f47-a33e-0db808e8b362/volumes"
Oct 31 08:07:43.743098 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743066    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:07:43.743375 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743361    4371 cpu_manager.go:414] "containerMap: removing container" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container"
Oct 31 08:07:43.743425 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743376    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container"
Oct 31 08:07:43.743425 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743401    4371 cpu_manager.go:414] "containerMap: removing container" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="storage-initializer"
Oct 31 08:07:43.743425 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743407    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="storage-initializer"
Oct 31 08:07:43.743425 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743419    4371 cpu_manager.go:414] "containerMap: removing container" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kube-rbac-proxy"
Oct 31 08:07:43.743425 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743423    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kube-rbac-proxy"
Oct 31 08:07:43.743618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743466    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kube-rbac-proxy"
Oct 31 08:07:43.743618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.743474    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f7dfe2-0b4f-4f47-a33e-0db808e8b362" containerName="kserve-container"
Oct 31 08:07:43.746358 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.746340    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.759152 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.759127    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:07:43.761398 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761378    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 31 08:07:43.761533 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761381    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:07:43.761603 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761399    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:07:43.761603 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761403    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 31 08:07:43.761603 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761428    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:07:43.761603 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761436    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:07:43.761603 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.761466    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:07:43.779716 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.779669    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:43.872917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.872880    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.872917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.872925    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.873149 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.872944    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cqq\" (UniqueName: \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.873149 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.873027    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.873222 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.873148    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.882854 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.882830    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:43.973764 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.973737    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.973933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.973780    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.973980 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.973915    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.973980 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.973964    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cqq\" (UniqueName: \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.974066 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.974001    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.974436 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.974417    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.974716 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.974695    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.974797 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.974738    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.976389 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.976367    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:43.982054 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:43.982035    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cqq\" (UniqueName: \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\") pod \"isvc-canary-promote-predictor-6bbfb747bd-rbx8s\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:44.061227 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:44.061190    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:44.330170 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:44.330145    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:07:44.332428 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:07:44.332395    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab87a430_f62b_4edb_8f5f_434848754aaa.slice/crio-ecd53620fcc64ad86594bbeeeb648e2c40f2ada31cb1423d0205e1c05870e014 WatchSource:0}: Error finding container ecd53620fcc64ad86594bbeeeb648e2c40f2ada31cb1423d0205e1c05870e014: Status 404 returned error can't find the container with id ecd53620fcc64ad86594bbeeeb648e2c40f2ada31cb1423d0205e1c05870e014
Oct 31 08:07:44.509460 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:44.509420    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:44.652226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:44.652138    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerStarted","Data":"686b5307f83fae7054dce5767764c73d1f6c59b6a1fea0bfd39ccab5f1595636"}
Oct 31 08:07:44.652226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:44.652195    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerStarted","Data":"ecd53620fcc64ad86594bbeeeb648e2c40f2ada31cb1423d0205e1c05870e014"}
Oct 31 08:07:48.670356 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:48.670319    4371 generic.go:356] "Generic (PLEG): container finished" podID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerID="686b5307f83fae7054dce5767764c73d1f6c59b6a1fea0bfd39ccab5f1595636" exitCode=0
Oct 31 08:07:48.670889 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:48.670395    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerDied","Data":"686b5307f83fae7054dce5767764c73d1f6c59b6a1fea0bfd39ccab5f1595636"}
Oct 31 08:07:49.677104 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:49.677070    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerStarted","Data":"49cf35b5860b1a247874fb7b4944116dd83eeed3ff78893d4b15fb015d855f13"}
Oct 31 08:07:49.677489 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:49.677132    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerStarted","Data":"0a34a1152597e53566fd33f9be80defcd652cc5e06e7ceb8663caf3fb3841bcb"}
Oct 31 08:07:49.677489 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:49.677375    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:49.696543 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:49.696480    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podStartSLOduration=2.648742883 podStartE2EDuration="6.696466912s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.047724029s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:49.695934113 +0000 UTC m=+2312.112064799" watchObservedRunningTime="2026-10-31 08:07:49.696466912 +0000 UTC m=+2312.112597583"
Oct 31 08:07:50.681764 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:50.681735    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:50.682895 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:50.682867    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:07:51.685274 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:51.685235    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:07:54.509449 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:54.509413    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:07:56.689885 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:56.689858    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:07:56.690357 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:07:56.690329    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:07:59.789194 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:07:59.789154    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:08:02.832385 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:02.832344    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:04.509662 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:04.509622    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:08:06.690910 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:06.690871    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:14.509988 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:14.509942    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:08:16.691280 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:16.691238    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:24.509659 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:24.509626    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:08:26.690274 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:26.690241    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:29.790576 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:08:29.790533    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:08:34.060646 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.060611    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:08:34.061149 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.060859    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" containerID="cri-o://73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1" gracePeriod=30
Oct 31 08:08:34.061149 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.060884    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kube-rbac-proxy" containerID="cri-o://b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946" gracePeriod=30
Oct 31 08:08:34.114186 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.114145    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:34.505144 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.505106    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Oct 31 08:08:34.509328 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.509302    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 31 08:08:34.856489 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.856403    4371 generic.go:356] "Generic (PLEG): container finished" podID="186deef1-25e7-4595-9932-31720a5f5c90" containerID="b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946" exitCode=0
Oct 31 08:08:34.856489 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:34.856473    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerDied","Data":"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"}
Oct 31 08:08:36.691294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:36.691255    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:38.398601 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.398576    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:08:38.511409 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511325    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\" (UniqueName: \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\") pod \"186deef1-25e7-4595-9932-31720a5f5c90\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") "
Oct 31 08:08:38.511409 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511377    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\") pod \"186deef1-25e7-4595-9932-31720a5f5c90\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") "
Oct 31 08:08:38.511409 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511404    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\" (UniqueName: \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\") pod \"186deef1-25e7-4595-9932-31720a5f5c90\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") "
Oct 31 08:08:38.511730 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511426    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\") pod \"186deef1-25e7-4595-9932-31720a5f5c90\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") "
Oct 31 08:08:38.511730 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511469    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\") pod \"186deef1-25e7-4595-9932-31720a5f5c90\" (UID: \"186deef1-25e7-4595-9932-31720a5f5c90\") "
Oct 31 08:08:38.511848 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511804    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location" pod "186deef1-25e7-4595-9932-31720a5f5c90" (UID: "186deef1-25e7-4595-9932-31720a5f5c90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:08:38.511887 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511842    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert" pod "186deef1-25e7-4595-9932-31720a5f5c90" (UID: "186deef1-25e7-4595-9932-31720a5f5c90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:08:38.511921 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.511858    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config" pod "186deef1-25e7-4595-9932-31720a5f5c90" (UID: "186deef1-25e7-4595-9932-31720a5f5c90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:08:38.513481 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.513456    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls" pod "186deef1-25e7-4595-9932-31720a5f5c90" (UID: "186deef1-25e7-4595-9932-31720a5f5c90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:08:38.513637 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.513616    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq" pod "186deef1-25e7-4595-9932-31720a5f5c90" (UID: "186deef1-25e7-4595-9932-31720a5f5c90"). InnerVolumeSpecName "kube-api-access-vb5jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:08:38.612680 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.612643    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186deef1-25e7-4595-9932-31720a5f5c90-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:08:38.612680 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.612672    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186deef1-25e7-4595-9932-31720a5f5c90-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:08:38.612680 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.612683    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:08:38.612680 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.612691    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186deef1-25e7-4595-9932-31720a5f5c90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:08:38.612940 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.612700    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5jq\" (UniqueName: \"kubernetes.io/projected/186deef1-25e7-4595-9932-31720a5f5c90-kube-api-access-vb5jq\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:08:38.873757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.873671    4371 generic.go:356] "Generic (PLEG): container finished" podID="186deef1-25e7-4595-9932-31720a5f5c90" containerID="73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1" exitCode=0
Oct 31 08:08:38.873757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.873723    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerDied","Data":"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"}
Oct 31 08:08:38.873757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.873736    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"
Oct 31 08:08:38.873983 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.873786    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5" event={"ID":"186deef1-25e7-4595-9932-31720a5f5c90","Type":"ContainerDied","Data":"6206a61616f0c84b57921d5a4f5e82dc0e5fbf982d4cd8bf5688620286356962"}
Oct 31 08:08:38.873983 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.873823    4371 scope.go:122] "RemoveContainer" containerID="b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"
Oct 31 08:08:38.882654 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.882618    4371 scope.go:122] "RemoveContainer" containerID="73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"
Oct 31 08:08:38.890475 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.890458    4371 scope.go:122] "RemoveContainer" containerID="d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"
Oct 31 08:08:38.895996 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.895973    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:08:38.898054 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.897959    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-predictor-75fd7f49d4-mcxf5"]
Oct 31 08:08:38.898117 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898086    4371 scope.go:122] "RemoveContainer" containerID="b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"
Oct 31 08:08:38.898362 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:08:38.898345    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946\": container with ID starting with b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946 not found: ID does not exist" containerID="b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"
Oct 31 08:08:38.898408 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898370    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946"} err="failed to get container status \"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946\": rpc error: code = NotFound desc = could not find container \"b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946\": container with ID starting with b75a40b7d71bf85e6f4dfd0a7c1e76eecc617612948791118f19ea5652856946 not found: ID does not exist"
Oct 31 08:08:38.898408 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898386    4371 scope.go:122] "RemoveContainer" containerID="73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"
Oct 31 08:08:38.898603 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:08:38.898585    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1\": container with ID starting with 73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1 not found: ID does not exist" containerID="73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"
Oct 31 08:08:38.898650 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898606    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1"} err="failed to get container status \"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1\": rpc error: code = NotFound desc = could not find container \"73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1\": container with ID starting with 73306dcdc87d98e8e219aacddbded355a024e04cbadfb6e297da7136b0c4e3a1 not found: ID does not exist"
Oct 31 08:08:38.898650 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898617    4371 scope.go:122] "RemoveContainer" containerID="d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"
Oct 31 08:08:38.898877 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:08:38.898858    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284\": container with ID starting with d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284 not found: ID does not exist" containerID="d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"
Oct 31 08:08:38.898926 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:38.898881    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284"} err="failed to get container status \"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284\": rpc error: code = NotFound desc = could not find container \"d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284\": container with ID starting with d7a283ce9f717f02451a9ae3db8500d9102475778d3e1ba854a939a0ee3ad284 not found: ID does not exist"
Oct 31 08:08:39.491230 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.491201    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:39.505258 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505230    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:08:39.505581 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505568    4371 cpu_manager.go:414] "containerMap: removing container" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container"
Oct 31 08:08:39.505643 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505583    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container"
Oct 31 08:08:39.505643 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505595    4371 cpu_manager.go:414] "containerMap: removing container" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="storage-initializer"
Oct 31 08:08:39.505643 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505600    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="storage-initializer"
Oct 31 08:08:39.505643 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505619    4371 cpu_manager.go:414] "containerMap: removing container" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kube-rbac-proxy"
Oct 31 08:08:39.505643 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505627    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kube-rbac-proxy"
Oct 31 08:08:39.505792 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505669    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kserve-container"
Oct 31 08:08:39.505792 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.505679    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="186deef1-25e7-4595-9932-31720a5f5c90" containerName="kube-rbac-proxy"
Oct 31 08:08:39.510221 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.510174    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.519554 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519500    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:08:39.519685 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519568    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:08:39.519685 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519590    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:08:39.519846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519742    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:08:39.519846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519752    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:08:39.519846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519776    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 31 08:08:39.519846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.519794    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 31 08:08:39.520075 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.520053    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:08:39.590852 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.590825    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:39.621965 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.621936    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.622073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.621980    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.622073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.622007    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdq5\" (UniqueName: \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.622180 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.622158    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.622218 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.622194    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.704613 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.704583    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186deef1-25e7-4595-9932-31720a5f5c90" path="/var/lib/kubelet/pods/186deef1-25e7-4595-9932-31720a5f5c90/volumes"
Oct 31 08:08:39.723343 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723318    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.723406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723346    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdq5\" (UniqueName: \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.723406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723399    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.723580 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723421    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.723580 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723458    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.723797 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.723770    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.724170 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.724148    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.724213 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.724155    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.725956 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.725933    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.731967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.731945    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdq5\" (UniqueName: \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\") pod \"isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:39.829224 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:39.829137    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:40.104458 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:40.104431    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:08:40.106314 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:08:40.106286    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fb45cb_c0da_4ff3_935a_f96a37f7dfdb.slice/crio-8a94c4fd8c0777354a5e6a9f330b0e467229df693a50bb3df49b391e7b968d3c WatchSource:0}: Error finding container 8a94c4fd8c0777354a5e6a9f330b0e467229df693a50bb3df49b391e7b968d3c: Status 404 returned error can't find the container with id 8a94c4fd8c0777354a5e6a9f330b0e467229df693a50bb3df49b391e7b968d3c
Oct 31 08:08:40.894034 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:40.893984    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerStarted","Data":"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"}
Oct 31 08:08:40.894577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:40.894055    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerStarted","Data":"8a94c4fd8c0777354a5e6a9f330b0e467229df693a50bb3df49b391e7b968d3c"}
Oct 31 08:08:44.911943 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:44.911908    4371 generic.go:356] "Generic (PLEG): container finished" podID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerID="e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21" exitCode=0
Oct 31 08:08:44.912342 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:44.911976    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerDied","Data":"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"}
Oct 31 08:08:45.918774 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:45.918740    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerStarted","Data":"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"}
Oct 31 08:08:45.919243 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:45.918801    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerStarted","Data":"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"}
Oct 31 08:08:45.919243 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:45.918985    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:45.941807 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:45.941748    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" podStartSLOduration=2.842490975 podStartE2EDuration="6.941732055s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.09924108s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:08:45.939563143 +0000 UTC m=+2368.355693815" watchObservedRunningTime="2026-10-31 08:08:45.941732055 +0000 UTC m=+2368.357862727"
Oct 31 08:08:46.690606 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:46.690550    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:46.923568 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:46.923538    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:52.932945 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:52.932915    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:08:56.691257 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:08:56.691218    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:08:59.789065 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:08:59.789028    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:09:02.883545 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:02.883468    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:06.690310 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:06.690268    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 31 08:09:16.691106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:16.691079    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:09:16.737075 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:16.737045    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:09:17.044060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:17.044021    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container" containerID="cri-o://0a34a1152597e53566fd33f9be80defcd652cc5e06e7ceb8663caf3fb3841bcb" gracePeriod=30
Oct 31 08:09:17.044060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:17.044049    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kube-rbac-proxy" containerID="cri-o://49cf35b5860b1a247874fb7b4944116dd83eeed3ff78893d4b15fb015d855f13" gracePeriod=30
Oct 31 08:09:18.049868 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:18.049833    4371 generic.go:356] "Generic (PLEG): container finished" podID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerID="49cf35b5860b1a247874fb7b4944116dd83eeed3ff78893d4b15fb015d855f13" exitCode=0
Oct 31 08:09:18.050259 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:18.049906    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerDied","Data":"49cf35b5860b1a247874fb7b4944116dd83eeed3ff78893d4b15fb015d855f13"}
Oct 31 08:09:21.685480 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:21.685435    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Oct 31 08:09:22.069559 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.069521    4371 generic.go:356] "Generic (PLEG): container finished" podID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerID="0a34a1152597e53566fd33f9be80defcd652cc5e06e7ceb8663caf3fb3841bcb" exitCode=0
Oct 31 08:09:22.069559 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.069537    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerDied","Data":"0a34a1152597e53566fd33f9be80defcd652cc5e06e7ceb8663caf3fb3841bcb"}
Oct 31 08:09:22.089235 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.089212    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:09:22.172952 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.172917    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\" (UniqueName: \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\") pod \"ab87a430-f62b-4edb-8f5f-434848754aaa\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") "
Oct 31 08:09:22.173132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.172959    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\") pod \"ab87a430-f62b-4edb-8f5f-434848754aaa\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") "
Oct 31 08:09:22.173132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173006    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\") pod \"ab87a430-f62b-4edb-8f5f-434848754aaa\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") "
Oct 31 08:09:22.173132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173037    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\") pod \"ab87a430-f62b-4edb-8f5f-434848754aaa\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") "
Oct 31 08:09:22.173132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173066    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\") pod \"ab87a430-f62b-4edb-8f5f-434848754aaa\" (UID: \"ab87a430-f62b-4edb-8f5f-434848754aaa\") "
Oct 31 08:09:22.173401 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173365    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location" pod "ab87a430-f62b-4edb-8f5f-434848754aaa" (UID: "ab87a430-f62b-4edb-8f5f-434848754aaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:22.173459 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173375    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config" pod "ab87a430-f62b-4edb-8f5f-434848754aaa" (UID: "ab87a430-f62b-4edb-8f5f-434848754aaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:22.173459 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.173414    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert" pod "ab87a430-f62b-4edb-8f5f-434848754aaa" (UID: "ab87a430-f62b-4edb-8f5f-434848754aaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:22.175056 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.175037    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq" pod "ab87a430-f62b-4edb-8f5f-434848754aaa" (UID: "ab87a430-f62b-4edb-8f5f-434848754aaa"). InnerVolumeSpecName "kube-api-access-l8cqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:09:22.175142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.175108    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls" pod "ab87a430-f62b-4edb-8f5f-434848754aaa" (UID: "ab87a430-f62b-4edb-8f5f-434848754aaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:09:22.274410 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.274376    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:22.274410 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.274408    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab87a430-f62b-4edb-8f5f-434848754aaa-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:22.274633 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.274419    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab87a430-f62b-4edb-8f5f-434848754aaa-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:22.274633 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.274428    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cqq\" (UniqueName: \"kubernetes.io/projected/ab87a430-f62b-4edb-8f5f-434848754aaa-kube-api-access-l8cqq\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:22.274633 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.274437    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab87a430-f62b-4edb-8f5f-434848754aaa-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:22.937407 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:22.937376    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:09:23.075099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.075064    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s" event={"ID":"ab87a430-f62b-4edb-8f5f-434848754aaa","Type":"ContainerDied","Data":"ecd53620fcc64ad86594bbeeeb648e2c40f2ada31cb1423d0205e1c05870e014"}
Oct 31 08:09:23.075300 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.075131    4371 scope.go:122] "RemoveContainer" containerID="49cf35b5860b1a247874fb7b4944116dd83eeed3ff78893d4b15fb015d855f13"
Oct 31 08:09:23.075300 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.075079    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"
Oct 31 08:09:23.083702 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.083682    4371 scope.go:122] "RemoveContainer" containerID="0a34a1152597e53566fd33f9be80defcd652cc5e06e7ceb8663caf3fb3841bcb"
Oct 31 08:09:23.090969 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.090952    4371 scope.go:122] "RemoveContainer" containerID="686b5307f83fae7054dce5767764c73d1f6c59b6a1fea0bfd39ccab5f1595636"
Oct 31 08:09:23.099135 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.099116    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:09:23.102606 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.102575    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-rbx8s"]
Oct 31 08:09:23.703853 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:23.703823    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" path="/var/lib/kubelet/pods/ab87a430-f62b-4edb-8f5f-434848754aaa/volumes"
Oct 31 08:09:24.007514 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007476    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007790    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kube-rbac-proxy"
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007800    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kube-rbac-proxy"
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007813    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container"
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007820    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container"
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007830    4371 cpu_manager.go:414] "containerMap: removing container" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="storage-initializer"
Oct 31 08:09:24.007879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007835    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="storage-initializer"
Oct 31 08:09:24.008062 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007898    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kube-rbac-proxy"
Oct 31 08:09:24.008062 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.007906    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87a430-f62b-4edb-8f5f-434848754aaa" containerName="kserve-container"
Oct 31 08:09:24.012496 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.012479    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.019887 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.019850    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:09:24.019887 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.019870    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:09:24.020168 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.019957    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:09:24.020168 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.020040    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 31 08:09:24.020168 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.020081    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:09:24.020324 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.020176    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 31 08:09:24.020440 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.020413    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:09:24.021393 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.021370    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:09:24.039798 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.039777    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:24.090352 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.090323    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.090482 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.090360    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.090482 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.090389    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmr5\" (UniqueName: \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.090482 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.090437    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.090640 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.090575    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.191714 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.191681    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.191880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.191734    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.191880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.191801    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.191880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.191846    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmr5\" (UniqueName: \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.191880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.191871    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.192296 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.192275    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.192560 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.192539    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.192560 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.192551    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.194164 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.194144    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.199572 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.199549    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmr5\" (UniqueName: \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-9vhfg\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.330791 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.330703    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:24.600403 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:24.600344    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:09:24.602746 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:09:24.602717    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c228e4f_8d64_4a74_b42a_c9e61780c6da.slice/crio-5ecaf979cf5d6789ef7733261b6bc8be164bb0de204faacfa508450774caf445 WatchSource:0}: Error finding container 5ecaf979cf5d6789ef7733261b6bc8be164bb0de204faacfa508450774caf445: Status 404 returned error can't find the container with id 5ecaf979cf5d6789ef7733261b6bc8be164bb0de204faacfa508450774caf445
Oct 31 08:09:25.086024 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:25.085987    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerStarted","Data":"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"}
Oct 31 08:09:25.086413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:25.086062    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerStarted","Data":"5ecaf979cf5d6789ef7733261b6bc8be164bb0de204faacfa508450774caf445"}
Oct 31 08:09:29.110113 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.110081    4371 generic.go:356] "Generic (PLEG): container finished" podID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerID="f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839" exitCode=0
Oct 31 08:09:29.110485 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.110155    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerDied","Data":"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"}
Oct 31 08:09:29.781206 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.781173    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:09:29.781456 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.781430    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kserve-container" containerID="cri-o://1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1" gracePeriod=30
Oct 31 08:09:29.781558 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.781459    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kube-rbac-proxy" containerID="cri-o://ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1" gracePeriod=30
Oct 31 08:09:29.801040 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:09:29.801012    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:09:29.843349 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:29.843313    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:30.116432 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.116352    4371 generic.go:356] "Generic (PLEG): container finished" podID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerID="ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1" exitCode=0
Oct 31 08:09:30.116432 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.116423    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerDied","Data":"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"}
Oct 31 08:09:30.118303 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.118280    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerStarted","Data":"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"}
Oct 31 08:09:30.118435 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.118338    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerStarted","Data":"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"}
Oct 31 08:09:30.118730 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.118707    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:30.118821 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.118755    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:30.120132 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.120107    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:09:30.139138 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:30.139084    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podStartSLOduration=3.07894145 podStartE2EDuration="7.139066506s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.060125056s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:09:30.135939762 +0000 UTC m=+2412.552070455" watchObservedRunningTime="2026-10-31 08:09:30.139066506 +0000 UTC m=+2412.555197183"
Oct 31 08:09:31.125585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:31.125543    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:09:32.927410 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:32.927368    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Oct 31 08:09:32.934001 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:32.933960    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 31 08:09:36.129989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.129954    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:09:36.130622 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.130594    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:09:36.323319 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.323294    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:09:36.396191 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396105    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\") pod \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") "
Oct 31 08:09:36.396191 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396154    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\" (UniqueName: \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\") pod \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") "
Oct 31 08:09:36.396406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396224    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\") pod \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") "
Oct 31 08:09:36.396406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396249    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\") pod \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") "
Oct 31 08:09:36.396406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396293    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\") pod \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\" (UID: \"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb\") "
Oct 31 08:09:36.396628 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396587    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location" pod "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" (UID: "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:36.396628 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396528    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert" pod "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" (UID: "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:36.396748 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.396705    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config" pod "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" (UID: "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:36.398201 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.398183    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5" pod "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" (UID: "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb"). InnerVolumeSpecName "kube-api-access-kwdq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:09:36.398313 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.398277    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls" pod "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" (UID: "e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:09:36.496901 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.496870    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:36.496901 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.496896    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdq5\" (UniqueName: \"kubernetes.io/projected/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-api-access-kwdq5\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:36.496901 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.496906    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:36.497121 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.496915    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:36.497121 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:36.496924    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:37.153330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.153295    4371 generic.go:356] "Generic (PLEG): container finished" podID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerID="1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1" exitCode=0
Oct 31 08:09:37.153754 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.153354    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"
Oct 31 08:09:37.153754 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.153371    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerDied","Data":"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"}
Oct 31 08:09:37.153754 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.153431    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4" event={"ID":"e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb","Type":"ContainerDied","Data":"8a94c4fd8c0777354a5e6a9f330b0e467229df693a50bb3df49b391e7b968d3c"}
Oct 31 08:09:37.153754 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.153457    4371 scope.go:122] "RemoveContainer" containerID="ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"
Oct 31 08:09:37.162049 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.162034    4371 scope.go:122] "RemoveContainer" containerID="1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"
Oct 31 08:09:37.169703 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.169687    4371 scope.go:122] "RemoveContainer" containerID="e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"
Oct 31 08:09:37.175235 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.175210    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:09:37.177093 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.177077    4371 scope.go:122] "RemoveContainer" containerID="ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"
Oct 31 08:09:37.177830 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:09:37.177802    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1\": container with ID starting with ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1 not found: ID does not exist" containerID="ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"
Oct 31 08:09:37.177973 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.177840    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1"} err="failed to get container status \"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1\": rpc error: code = NotFound desc = could not find container \"ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1\": container with ID starting with ef00f3171a1159cd28e38c7e9528137afde9a08f1f33121027d969966cabdca1 not found: ID does not exist"
Oct 31 08:09:37.177973 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.177861    4371 scope.go:122] "RemoveContainer" containerID="1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"
Oct 31 08:09:37.178178 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:09:37.178161    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1\": container with ID starting with 1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1 not found: ID does not exist" containerID="1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"
Oct 31 08:09:37.178226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.178185    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1"} err="failed to get container status \"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1\": rpc error: code = NotFound desc = could not find container \"1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1\": container with ID starting with 1f6e76a125c56603c4730045927d4f66c15df835988e1482ddff2dd3eb866ab1 not found: ID does not exist"
Oct 31 08:09:37.178226 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.178200    4371 scope.go:122] "RemoveContainer" containerID="e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"
Oct 31 08:09:37.178464 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:09:37.178446    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21\": container with ID starting with e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21 not found: ID does not exist" containerID="e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"
Oct 31 08:09:37.178504 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.178471    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21"} err="failed to get container status \"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21\": rpc error: code = NotFound desc = could not find container \"e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21\": container with ID starting with e3db2b3043ae5e5742cf55932bdaec8ec89412463ba1be89020fe1d2aa06db21 not found: ID does not exist"
Oct 31 08:09:37.179196 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.179178    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-mlserver-predictor-64f44b797d-m6bt4"]
Oct 31 08:09:37.703353 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:37.703323    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" path="/var/lib/kubelet/pods/e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb/volumes"
Oct 31 08:09:39.275904 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.275875    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:39.283558 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283533    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:09:39.283886 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283872    4371 cpu_manager.go:414] "containerMap: removing container" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kserve-container"
Oct 31 08:09:39.283933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283886    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kserve-container"
Oct 31 08:09:39.283933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283896    4371 cpu_manager.go:414] "containerMap: removing container" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="storage-initializer"
Oct 31 08:09:39.283933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283901    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="storage-initializer"
Oct 31 08:09:39.283933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283911    4371 cpu_manager.go:414] "containerMap: removing container" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kube-rbac-proxy"
Oct 31 08:09:39.283933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283916    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kube-rbac-proxy"
Oct 31 08:09:39.284093 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.283991    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kserve-container"
Oct 31 08:09:39.284093 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.284000    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fb45cb-c0da-4ff3-935a-f96a37f7dfdb" containerName="kube-rbac-proxy"
Oct 31 08:09:39.288695 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.288677    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.294361 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.294340    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 31 08:09:39.297240 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.297213    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.299234    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.300161    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.300314    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.300418    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.300434    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:09:39.303594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.300501    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 31 08:09:39.379260 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.379226    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:39.422408 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.422377    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfrk\" (UniqueName: \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.422559 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.422424    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.422559 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.422445    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.422679 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.422607    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.422723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.422691    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.523740 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.523692    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfrk\" (UniqueName: \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.523911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.523757    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.523911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.523781    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.523911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.523823    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.523911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.523854    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.524261 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.524236    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.524693 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.524669    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.524741 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.524669    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.526399 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.526356    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.531911 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.531888    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfrk\" (UniqueName: \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\") pod \"xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") " pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.614683 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.614645    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:39.887170 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:39.887141    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:09:39.889341 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:09:39.889310    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3662590f_f6f0_440b_92c4_c21107bc6152.slice/crio-2377491a40e9ba07e589c2f42eb3ae52a86f4a76d23e237d0c3319dc70af8312 WatchSource:0}: Error finding container 2377491a40e9ba07e589c2f42eb3ae52a86f4a76d23e237d0c3319dc70af8312: Status 404 returned error can't find the container with id 2377491a40e9ba07e589c2f42eb3ae52a86f4a76d23e237d0c3319dc70af8312
Oct 31 08:09:40.167848 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:40.167761    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerStarted","Data":"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"}
Oct 31 08:09:40.167848 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:40.167823    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerStarted","Data":"2377491a40e9ba07e589c2f42eb3ae52a86f4a76d23e237d0c3319dc70af8312"}
Oct 31 08:09:44.185922 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:44.185836    4371 generic.go:356] "Generic (PLEG): container finished" podID="3662590f-f6f0-440b-92c4-c21107bc6152" containerID="92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457" exitCode=0
Oct 31 08:09:44.185922 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:44.185874    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerDied","Data":"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"}
Oct 31 08:09:45.192312 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:45.192278    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerStarted","Data":"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"}
Oct 31 08:09:45.192732 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:45.192341    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerStarted","Data":"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"}
Oct 31 08:09:45.192732 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:45.192534    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:45.213804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:45.213749    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" podStartSLOduration=2.217704002 podStartE2EDuration="6.213737439s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996033437s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:09:45.211725644 +0000 UTC m=+2427.627856331" watchObservedRunningTime="2026-10-31 08:09:45.213737439 +0000 UTC m=+2427.629868110"
Oct 31 08:09:46.131253 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:46.131213    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:09:46.196829 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:46.196798    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:52.206067 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:52.206039    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:09:56.130635 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:09:56.130591    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:09:59.789627 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:09:59.789592    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:10:02.930188 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:02.930144    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:06.131259 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:06.131210    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:10:16.130992 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:16.130949    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:10:22.209870 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:22.209841    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:10:26.130856 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:26.130814    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:10:29.606492 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:29.606453    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:10:29.606891 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:29.606820    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kserve-container" containerID="cri-o://bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39" gracePeriod=30
Oct 31 08:10:29.606891 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:29.606865    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kube-rbac-proxy" containerID="cri-o://3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104" gracePeriod=30
Oct 31 08:10:29.664178 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:29.664137    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:29.789617 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:10:29.789582    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:10:30.379045 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:30.379006    4371 generic.go:356] "Generic (PLEG): container finished" podID="3662590f-f6f0-440b-92c4-c21107bc6152" containerID="3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104" exitCode=0
Oct 31 08:10:30.379218 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:30.379061    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerDied","Data":"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"}
Oct 31 08:10:32.201131 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:32.201040    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Oct 31 08:10:32.206561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:32.206532    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 31 08:10:36.130776 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.130727    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:10:36.150227 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.150205    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:10:36.297750 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.297697    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\") pod \"3662590f-f6f0-440b-92c4-c21107bc6152\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") "
Oct 31 08:10:36.297929 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.297807    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\" (UniqueName: \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\") pod \"3662590f-f6f0-440b-92c4-c21107bc6152\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") "
Oct 31 08:10:36.297929 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.297854    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\") pod \"3662590f-f6f0-440b-92c4-c21107bc6152\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") "
Oct 31 08:10:36.297929 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.297893    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\") pod \"3662590f-f6f0-440b-92c4-c21107bc6152\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") "
Oct 31 08:10:36.298111 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.297930    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\") pod \"3662590f-f6f0-440b-92c4-c21107bc6152\" (UID: \"3662590f-f6f0-440b-92c4-c21107bc6152\") "
Oct 31 08:10:36.298216 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.298187    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config" pod "3662590f-f6f0-440b-92c4-c21107bc6152" (UID: "3662590f-f6f0-440b-92c4-c21107bc6152"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:10:36.298321 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.298277    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location" pod "3662590f-f6f0-440b-92c4-c21107bc6152" (UID: "3662590f-f6f0-440b-92c4-c21107bc6152"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:10:36.298384 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.298333    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert" pod "3662590f-f6f0-440b-92c4-c21107bc6152" (UID: "3662590f-f6f0-440b-92c4-c21107bc6152"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:10:36.299967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.299946    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk" pod "3662590f-f6f0-440b-92c4-c21107bc6152" (UID: "3662590f-f6f0-440b-92c4-c21107bc6152"). InnerVolumeSpecName "kube-api-access-xlfrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:10:36.300049 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.299978    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls" pod "3662590f-f6f0-440b-92c4-c21107bc6152" (UID: "3662590f-f6f0-440b-92c4-c21107bc6152"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:10:36.399407 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.399375    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:10:36.399407 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.399404    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlfrk\" (UniqueName: \"kubernetes.io/projected/3662590f-f6f0-440b-92c4-c21107bc6152-kube-api-access-xlfrk\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:10:36.399407 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.399416    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3662590f-f6f0-440b-92c4-c21107bc6152-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:10:36.399668 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.399424    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3662590f-f6f0-440b-92c4-c21107bc6152-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:10:36.399668 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.399435    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3662590f-f6f0-440b-92c4-c21107bc6152-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:10:36.404599 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.404561    4371 generic.go:356] "Generic (PLEG): container finished" podID="3662590f-f6f0-440b-92c4-c21107bc6152" containerID="bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39" exitCode=0
Oct 31 08:10:36.404735 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.404620    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"
Oct 31 08:10:36.404735 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.404628    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerDied","Data":"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"}
Oct 31 08:10:36.404735 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.404679    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54" event={"ID":"3662590f-f6f0-440b-92c4-c21107bc6152","Type":"ContainerDied","Data":"2377491a40e9ba07e589c2f42eb3ae52a86f4a76d23e237d0c3319dc70af8312"}
Oct 31 08:10:36.404735 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.404706    4371 scope.go:122] "RemoveContainer" containerID="3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"
Oct 31 08:10:36.415694 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.415676    4371 scope.go:122] "RemoveContainer" containerID="bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"
Oct 31 08:10:36.423148 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.423131    4371 scope.go:122] "RemoveContainer" containerID="92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"
Oct 31 08:10:36.428868 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.428844    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:10:36.430358 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.430341    4371 scope.go:122] "RemoveContainer" containerID="3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"
Oct 31 08:10:36.430683 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:10:36.430656    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104\": container with ID starting with 3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104 not found: ID does not exist" containerID="3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"
Oct 31 08:10:36.430762 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.430697    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104"} err="failed to get container status \"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104\": rpc error: code = NotFound desc = could not find container \"3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104\": container with ID starting with 3e99a8201f0c012bd7ce01ffc1ff163e7710b6afc254f1bbb8d6d773b002d104 not found: ID does not exist"
Oct 31 08:10:36.430762 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.430720    4371 scope.go:122] "RemoveContainer" containerID="bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"
Oct 31 08:10:36.430975 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:10:36.430959    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39\": container with ID starting with bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39 not found: ID does not exist" containerID="bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"
Oct 31 08:10:36.431027 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.430983    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39"} err="failed to get container status \"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39\": rpc error: code = NotFound desc = could not find container \"bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39\": container with ID starting with bff9ac9b11cb4491ca57e65cce84a02184d7d1dd033fac31f06fa932a6de5a39 not found: ID does not exist"
Oct 31 08:10:36.431027 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.430998    4371 scope.go:122] "RemoveContainer" containerID="92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"
Oct 31 08:10:36.431210 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:10:36.431195    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457\": container with ID starting with 92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457 not found: ID does not exist" containerID="92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"
Oct 31 08:10:36.431253 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.431213    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457"} err="failed to get container status \"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457\": rpc error: code = NotFound desc = could not find container \"92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457\": container with ID starting with 92b927b0a6462bfd1402a0b720557939f99414ca4a9b53702462fafa3a873457 not found: ID does not exist"
Oct 31 08:10:36.433883 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:36.433861    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/xgboost-v2-mlserver-predictor-77bcf9b7f5-fxr54"]
Oct 31 08:10:37.083691 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:37.083658    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:37.704112 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:37.704079    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" path="/var/lib/kubelet/pods/3662590f-f6f0-440b-92c4-c21107bc6152/volumes"
Oct 31 08:10:38.406014 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:38.405979    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:46.131187 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:46.131145    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:10:56.131706 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:10:56.131671    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:10:59.789829 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:10:59.789791    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:11:02.966596 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:02.966554    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:04.448018 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:04.447983    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:11:04.448470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:04.448254    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" containerID="cri-o://ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf" gracePeriod=30
Oct 31 08:11:04.448470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:04.448430    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kube-rbac-proxy" containerID="cri-o://d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1" gracePeriod=30
Oct 31 08:11:04.503144 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:04.503116    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:05.515498 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:05.515464    4371 generic.go:356] "Generic (PLEG): container finished" podID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerID="d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1" exitCode=0
Oct 31 08:11:05.515896 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:05.515538    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerDied","Data":"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"}
Oct 31 08:11:06.125824 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:06.125786    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Oct 31 08:11:06.130988 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:06.130957    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 31 08:11:09.386501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.386477    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:11:09.462914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.462885    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\" (UniqueName: \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\") pod \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") "
Oct 31 08:11:09.463099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.462950    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\") pod \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") "
Oct 31 08:11:09.463099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463000    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\") pod \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") "
Oct 31 08:11:09.463099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463045    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\") pod \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") "
Oct 31 08:11:09.463099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463092    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\") pod \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\" (UID: \"7c228e4f-8d64-4a74-b42a-c9e61780c6da\") "
Oct 31 08:11:09.463374 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463352    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config" pod "7c228e4f-8d64-4a74-b42a-c9e61780c6da" (UID: "7c228e4f-8d64-4a74-b42a-c9e61780c6da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:09.463437 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463408    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location" pod "7c228e4f-8d64-4a74-b42a-c9e61780c6da" (UID: "7c228e4f-8d64-4a74-b42a-c9e61780c6da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:09.463437 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.463420    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert" pod "7c228e4f-8d64-4a74-b42a-c9e61780c6da" (UID: "7c228e4f-8d64-4a74-b42a-c9e61780c6da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:09.465010 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.464989    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5" pod "7c228e4f-8d64-4a74-b42a-c9e61780c6da" (UID: "7c228e4f-8d64-4a74-b42a-c9e61780c6da"). InnerVolumeSpecName "kube-api-access-fqmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:11:09.465109 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.465059    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls" pod "7c228e4f-8d64-4a74-b42a-c9e61780c6da" (UID: "7c228e4f-8d64-4a74-b42a-c9e61780c6da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:11:09.532303 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.532235    4371 generic.go:356] "Generic (PLEG): container finished" podID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerID="ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf" exitCode=0
Oct 31 08:11:09.532303 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.532292    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"
Oct 31 08:11:09.532463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.532296    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerDied","Data":"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"}
Oct 31 08:11:09.532463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.532358    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg" event={"ID":"7c228e4f-8d64-4a74-b42a-c9e61780c6da","Type":"ContainerDied","Data":"5ecaf979cf5d6789ef7733261b6bc8be164bb0de204faacfa508450774caf445"}
Oct 31 08:11:09.532463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.532389    4371 scope.go:122] "RemoveContainer" containerID="d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"
Oct 31 08:11:09.541673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.541649    4371 scope.go:122] "RemoveContainer" containerID="ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"
Oct 31 08:11:09.549952 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.549936    4371 scope.go:122] "RemoveContainer" containerID="f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"
Oct 31 08:11:09.556033 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.556012    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:11:09.560476 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.560457    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-9vhfg"]
Oct 31 08:11:09.563313 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563296    4371 scope.go:122] "RemoveContainer" containerID="d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"
Oct 31 08:11:09.563596 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:11:09.563575    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1\": container with ID starting with d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1 not found: ID does not exist" containerID="d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"
Oct 31 08:11:09.563664 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563607    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1"} err="failed to get container status \"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1\": rpc error: code = NotFound desc = could not find container \"d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1\": container with ID starting with d6d74c86418ea8258d4b0c52a527990971efe36fb25b52ff4e4c161aeb1365d1 not found: ID does not exist"
Oct 31 08:11:09.563664 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563624    4371 scope.go:122] "RemoveContainer" containerID="ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"
Oct 31 08:11:09.563781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563581    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.563781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563712    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c228e4f-8d64-4a74-b42a-c9e61780c6da-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.563781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563730    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c228e4f-8d64-4a74-b42a-c9e61780c6da-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.563781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563746    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.563781 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563761    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmr5\" (UniqueName: \"kubernetes.io/projected/7c228e4f-8d64-4a74-b42a-c9e61780c6da-kube-api-access-fqmr5\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.563951 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:11:09.563829    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf\": container with ID starting with ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf not found: ID does not exist" containerID="ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"
Oct 31 08:11:09.563951 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563852    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf"} err="failed to get container status \"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf\": rpc error: code = NotFound desc = could not find container \"ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf\": container with ID starting with ede9e429ae0035469d5125b3a4a95e0500297d6923164751b97de35f423989bf not found: ID does not exist"
Oct 31 08:11:09.563951 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.563866    4371 scope.go:122] "RemoveContainer" containerID="f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"
Oct 31 08:11:09.564068 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:11:09.564041    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839\": container with ID starting with f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839 not found: ID does not exist" containerID="f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"
Oct 31 08:11:09.564068 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.564058    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839"} err="failed to get container status \"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839\": rpc error: code = NotFound desc = could not find container \"f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839\": container with ID starting with f9d7580dba33452c52c6b0b6bd3b91f7bec976d1bc0c0c37005f78cff7704839 not found: ID does not exist"
Oct 31 08:11:09.703026 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:09.702991    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" path="/var/lib/kubelet/pods/7c228e4f-8d64-4a74-b42a-c9e61780c6da/volumes"
Oct 31 08:11:11.975173 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975141    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975432    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="storage-initializer"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975442    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="storage-initializer"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975453    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975459    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975466    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kserve-container"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975471    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kserve-container"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975491    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="storage-initializer"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975496    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="storage-initializer"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975531    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kube-rbac-proxy"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975537    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kube-rbac-proxy"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975545    4371 cpu_manager.go:414] "containerMap: removing container" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kube-rbac-proxy"
Oct 31 08:11:11.975573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975550    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kube-rbac-proxy"
Oct 31 08:11:11.976189 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975597    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kube-rbac-proxy"
Oct 31 08:11:11.976189 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975604    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kube-rbac-proxy"
Oct 31 08:11:11.976189 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975612    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c228e4f-8d64-4a74-b42a-c9e61780c6da" containerName="kserve-container"
Oct 31 08:11:11.976189 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.975617    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="3662590f-f6f0-440b-92c4-c21107bc6152" containerName="kserve-container"
Oct 31 08:11:11.980337 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.980316    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:11.985772 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.985661    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 31 08:11:11.985772 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.985721    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:11:11.988162 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.987333    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:11:11.988162 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.987412    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:11:11.988162 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.987451    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:11:11.988162 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.987580    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 31 08:11:11.988162 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.987650    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:11:11.989294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:11.989271    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:11:12.001674 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.001618    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:12.072991 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.072961    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:11:12.076445 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.076430    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.081846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.081818    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8sf\" (UniqueName: \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.081992 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.081877    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.081992 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.081919    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.082121 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.082052    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.082121 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.082111    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.085728 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.085712    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 31 08:11:12.088746 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.088724    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:11:12.116227 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.116198    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:12.183271 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183240    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.183394 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183276    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8sf\" (UniqueName: \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.183394 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183313    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.183394 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183378    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.183498 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183439    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.183498 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183469    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gf8\" (UniqueName: \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.183690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183670    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.183761 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183723    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.183814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183763    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.183814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.183806    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.184101 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.184073    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.184168 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.184079    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.184269 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.184254    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.185802 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.185784    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.192127 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.192107    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8sf\" (UniqueName: \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\") pod \"isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.213225 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.213202    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:12.285087 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285014    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gf8\" (UniqueName: \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285087 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285054    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285087 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285076    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285348 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285100    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285348 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285135    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285580    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285735    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.285851 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.285830    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.287411 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.287394    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.293025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.293005    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gf8\" (UniqueName: \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-8stkn\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.299714 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.299694    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:12.394092 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.394063    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:12.571444 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.571415    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:11:12.573239 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:11:12.573214    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb982ab2a_6a40_419a_84fd_8883a9a1ef17.slice/crio-51d023a4d0440263f3a3d1a08d1f5467d742ba093b7e6cda9226a0597f15cd89 WatchSource:0}: Error finding container 51d023a4d0440263f3a3d1a08d1f5467d742ba093b7e6cda9226a0597f15cd89: Status 404 returned error can't find the container with id 51d023a4d0440263f3a3d1a08d1f5467d742ba093b7e6cda9226a0597f15cd89
Oct 31 08:11:12.668111 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:12.668090    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:11:12.670572 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:11:12.670540    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218a9d59_74da_40a4_8cb5_d107bdd0d70b.slice/crio-061cd8856bfe960212dbf470dd8ca9e9da7bc266d068658e92beab5b3c667b8d WatchSource:0}: Error finding container 061cd8856bfe960212dbf470dd8ca9e9da7bc266d068658e92beab5b3c667b8d: Status 404 returned error can't find the container with id 061cd8856bfe960212dbf470dd8ca9e9da7bc266d068658e92beab5b3c667b8d
Oct 31 08:11:13.550374 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:13.550338    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerStarted","Data":"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"}
Oct 31 08:11:13.550840 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:13.550397    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerStarted","Data":"51d023a4d0440263f3a3d1a08d1f5467d742ba093b7e6cda9226a0597f15cd89"}
Oct 31 08:11:13.551983 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:13.551960    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerStarted","Data":"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"}
Oct 31 08:11:13.552025 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:13.552005    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerStarted","Data":"061cd8856bfe960212dbf470dd8ca9e9da7bc266d068658e92beab5b3c667b8d"}
Oct 31 08:11:16.566720 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:16.566686    4371 generic.go:356] "Generic (PLEG): container finished" podID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerID="0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179" exitCode=0
Oct 31 08:11:16.567074 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:16.566759    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerDied","Data":"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"}
Oct 31 08:11:17.572852 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.572816    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerStarted","Data":"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"}
Oct 31 08:11:17.573305 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.572885    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerStarted","Data":"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"}
Oct 31 08:11:17.573305 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.573095    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:17.574356 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.574331    4371 generic.go:356] "Generic (PLEG): container finished" podID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerID="da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef" exitCode=0
Oct 31 08:11:17.574470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.574401    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerDied","Data":"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"}
Oct 31 08:11:17.593864 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:17.593817    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podStartSLOduration=2.736083229 podStartE2EDuration="6.593802836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857719607s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:17.592792766 +0000 UTC m=+2520.008923438" watchObservedRunningTime="2026-10-31 08:11:17.593802836 +0000 UTC m=+2520.009933507"
Oct 31 08:11:18.580417 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.580376    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerStarted","Data":"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"}
Oct 31 08:11:18.580820 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.580703    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:18.580820 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.580751    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerStarted","Data":"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"}
Oct 31 08:11:18.581056 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.581019    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:18.581201 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.581073    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:18.582011 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.581978    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:18.582102 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.582056    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:18.601895 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:18.601856    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podStartSLOduration=2.626289354 podStartE2EDuration="6.601845228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975555874s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:18.599996116 +0000 UTC m=+2521.016126788" watchObservedRunningTime="2026-10-31 08:11:18.601845228 +0000 UTC m=+2521.017975900"
Oct 31 08:11:19.584577 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:19.584536    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:19.585050 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:19.584662    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:24.588724 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:24.588694    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:11:24.589109 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:24.589061    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:11:24.589109 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:24.589092    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:24.589457 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:24.589438    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:29.788989 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:11:29.788946    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:11:34.589777 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:34.589740    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:34.590247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:34.589740    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:44.589134 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:44.589098    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:44.589534 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:44.589498    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:54.589960 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:54.589916    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:11:54.590437 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:11:54.589925    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:11:59.789872 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:11:59.789830    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:12:03.005700 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:03.005608    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:04.589152 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:04.589114    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:12:04.589563 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:04.589541    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:12:14.589463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:14.589422    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:12:14.589463 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:14.589441    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:12:18.023233 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:18.023196    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:23.416961 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.416926    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:12:23.420242 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.420225    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.430867 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.430842    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:12:23.430999 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.430985    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 31 08:12:23.431074 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.431061    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:12:23.431140 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.431107    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:12:23.431202 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.431166    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:12:23.431254 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.431222    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:12:23.431909 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.431888    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:12:23.432631 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.432609    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:12:23.443378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.437712    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:23.486977 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.486939    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.487142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.486988    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.487142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.487072    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksm6r\" (UniqueName: \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.487142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.487108    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.487142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.487140    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.535900 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.535854    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:23.588144 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588089    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588144 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588157    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588185    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksm6r\" (UniqueName: \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588203    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588402 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588358    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588774 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588753    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588898    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.588961 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.588928    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.590854 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.590836    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.596538 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.596492    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksm6r\" (UniqueName: \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\") pod \"isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:23.745396 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:23.745353    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:24.017796 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:24.017770    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:12:24.019999 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:12:24.019970    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df63a7d_5b91_49f9_811b_ade411aaf0a4.slice/crio-84403eb261f56e698184c68d19912987c46611fffc78564629e5a699822ddfd8 WatchSource:0}: Error finding container 84403eb261f56e698184c68d19912987c46611fffc78564629e5a699822ddfd8: Status 404 returned error can't find the container with id 84403eb261f56e698184c68d19912987c46611fffc78564629e5a699822ddfd8
Oct 31 08:12:24.589182 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:24.589145    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:12:24.589655 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:24.589619    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:12:24.839938 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:24.839843    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerStarted","Data":"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"}
Oct 31 08:12:24.839938 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:24.839906    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerStarted","Data":"84403eb261f56e698184c68d19912987c46611fffc78564629e5a699822ddfd8"}
Oct 31 08:12:27.701820 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:27.701779    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 31 08:12:28.856617 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:28.856583    4371 generic.go:356] "Generic (PLEG): container finished" podID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerID="d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934" exitCode=0
Oct 31 08:12:28.857001 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:28.856652    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerDied","Data":"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"}
Oct 31 08:12:29.789810 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:12:29.789770    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:12:29.863037 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:29.863000    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerStarted","Data":"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"}
Oct 31 08:12:29.863564 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:29.863068    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerStarted","Data":"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"}
Oct 31 08:12:29.863564 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:29.863280    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:29.885589 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:29.885540    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" podStartSLOduration=2.8133508000000003 podStartE2EDuration="6.88552712s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.07217632s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:12:29.882895377 +0000 UTC m=+2592.299026048" watchObservedRunningTime="2026-10-31 08:12:29.88552712 +0000 UTC m=+2592.301657791"
Oct 31 08:12:30.871989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:30.871955    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:34.590204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:34.590160    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:12:36.882381 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:36.882346    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:12:37.704266 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:37.704237    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:12:44.590673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:44.590637    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:12:52.150172 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.150133    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:12:52.150772 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.150380    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container" containerID="cri-o://3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676" gracePeriod=30
Oct 31 08:12:52.150772 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.150427    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kube-rbac-proxy" containerID="cri-o://105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa" gracePeriod=30
Oct 31 08:12:52.250711 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.250674    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:52.964661 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.964626    4371 generic.go:356] "Generic (PLEG): container finished" podID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerID="105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa" exitCode=0
Oct 31 08:12:52.964834 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:52.964693    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerDied","Data":"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"}
Oct 31 08:12:54.585209 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:54.585166    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 31 08:12:56.893596 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.893571    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:12:56.976500 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976461    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\" (UniqueName: \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\") pod \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") "
Oct 31 08:12:56.976500 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976503    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\") pod \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") "
Oct 31 08:12:56.976734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976552    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\") pod \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") "
Oct 31 08:12:56.976734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976624    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\") pod \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") "
Oct 31 08:12:56.976734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976703    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\") pod \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\" (UID: \"b982ab2a-6a40-419a-84fd-8883a9a1ef17\") "
Oct 31 08:12:56.976918 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.976889    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert" pod "b982ab2a-6a40-419a-84fd-8883a9a1ef17" (UID: "b982ab2a-6a40-419a-84fd-8883a9a1ef17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:56.977063 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.977040    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location" pod "b982ab2a-6a40-419a-84fd-8883a9a1ef17" (UID: "b982ab2a-6a40-419a-84fd-8883a9a1ef17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:56.977120 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.977100    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config" pod "b982ab2a-6a40-419a-84fd-8883a9a1ef17" (UID: "b982ab2a-6a40-419a-84fd-8883a9a1ef17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:56.979150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.979126    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf" pod "b982ab2a-6a40-419a-84fd-8883a9a1ef17" (UID: "b982ab2a-6a40-419a-84fd-8883a9a1ef17"). InnerVolumeSpecName "kube-api-access-2j8sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:56.979252 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.979207    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls" pod "b982ab2a-6a40-419a-84fd-8883a9a1ef17" (UID: "b982ab2a-6a40-419a-84fd-8883a9a1ef17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:12:56.983804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.983670    4371 generic.go:356] "Generic (PLEG): container finished" podID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerID="3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676" exitCode=0
Oct 31 08:12:56.983804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.983750    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"
Oct 31 08:12:56.983804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.983747    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerDied","Data":"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"}
Oct 31 08:12:56.983914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.983838    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p" event={"ID":"b982ab2a-6a40-419a-84fd-8883a9a1ef17","Type":"ContainerDied","Data":"51d023a4d0440263f3a3d1a08d1f5467d742ba093b7e6cda9226a0597f15cd89"}
Oct 31 08:12:56.983914 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.983865    4371 scope.go:122] "RemoveContainer" containerID="105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"
Oct 31 08:12:56.992325 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.992304    4371 scope.go:122] "RemoveContainer" containerID="3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"
Oct 31 08:12:56.999566 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:56.999550    4371 scope.go:122] "RemoveContainer" containerID="0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"
Oct 31 08:12:57.007198 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007178    4371 scope.go:122] "RemoveContainer" containerID="105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"
Oct 31 08:12:57.007489 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:12:57.007469    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa\": container with ID starting with 105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa not found: ID does not exist" containerID="105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"
Oct 31 08:12:57.007561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007498    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa"} err="failed to get container status \"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa\": rpc error: code = NotFound desc = could not find container \"105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa\": container with ID starting with 105f87eb78c30ad204456d8b389b5afd3954b37e3fade935b4bc0f24918026fa not found: ID does not exist"
Oct 31 08:12:57.007561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007528    4371 scope.go:122] "RemoveContainer" containerID="3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"
Oct 31 08:12:57.007561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007537    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:12:57.007770 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:12:57.007755    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676\": container with ID starting with 3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676 not found: ID does not exist" containerID="3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"
Oct 31 08:12:57.007814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007774    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676"} err="failed to get container status \"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676\": rpc error: code = NotFound desc = could not find container \"3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676\": container with ID starting with 3d44c7e0309fa814e07fd5ff5863237ff3782654e541a458779a262589ca8676 not found: ID does not exist"
Oct 31 08:12:57.007814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.007786    4371 scope.go:122] "RemoveContainer" containerID="0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"
Oct 31 08:12:57.008000 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:12:57.007985    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179\": container with ID starting with 0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179 not found: ID does not exist" containerID="0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"
Oct 31 08:12:57.008040 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.008003    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179"} err="failed to get container status \"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179\": rpc error: code = NotFound desc = could not find container \"0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179\": container with ID starting with 0dac70e8954f4f0e71839fa0fe3858f11023614f70ead5a486b266dd60e4d179 not found: ID does not exist"
Oct 31 08:12:57.011043 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.011021    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-v2-predictor-55d7c84779-kfw8p"]
Oct 31 08:12:57.078143 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.078097    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:57.078143 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.078140    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:57.078143 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.078151    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8sf\" (UniqueName: \"kubernetes.io/projected/b982ab2a-6a40-419a-84fd-8883a9a1ef17-kube-api-access-2j8sf\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:57.078361 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.078163    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b982ab2a-6a40-419a-84fd-8883a9a1ef17-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:57.078361 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.078173    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b982ab2a-6a40-419a-84fd-8883a9a1ef17-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:57.703625 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:12:57.703595    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" path="/var/lib/kubelet/pods/b982ab2a-6a40-419a-84fd-8883a9a1ef17/volumes"
Oct 31 08:12:59.789955 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:12:59.789916    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:13:01.677179 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:01.677144    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:03.044780 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:03.044745    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:06.931468 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:06.931408    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 31 08:13:16.885406 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:16.885375    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:13:23.747195 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:23.747161    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:13:23.747790 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:23.747394    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kserve-container" containerID="cri-o://840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df" gracePeriod=30
Oct 31 08:13:23.747790 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:23.747477    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kube-rbac-proxy" containerID="cri-o://7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802" gracePeriod=30
Oct 31 08:13:23.806213 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:23.806183    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:24.094353 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:24.094267    4371 generic.go:356] "Generic (PLEG): container finished" podID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerID="7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802" exitCode=0
Oct 31 08:13:24.094353 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:24.094340    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerDied","Data":"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"}
Oct 31 08:13:26.877231 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:26.877187    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Oct 31 08:13:29.789389 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:13:29.789348    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:13:31.293003 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.292971    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:13:31.473954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.473927    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\") pod \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") "
Oct 31 08:13:31.474263 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474005    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\") pod \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") "
Oct 31 08:13:31.474263 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474032    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\") pod \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") "
Oct 31 08:13:31.474263 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474051    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\" (UniqueName: \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\") pod \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") "
Oct 31 08:13:31.474263 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474090    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\") pod \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\" (UID: \"6df63a7d-5b91-49f9-811b-ade411aaf0a4\") "
Oct 31 08:13:31.474492 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474287    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location" pod "6df63a7d-5b91-49f9-811b-ade411aaf0a4" (UID: "6df63a7d-5b91-49f9-811b-ade411aaf0a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:13:31.474492 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474439    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config" pod "6df63a7d-5b91-49f9-811b-ade411aaf0a4" (UID: "6df63a7d-5b91-49f9-811b-ade411aaf0a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:13:31.474595 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.474537    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert" pod "6df63a7d-5b91-49f9-811b-ade411aaf0a4" (UID: "6df63a7d-5b91-49f9-811b-ade411aaf0a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:13:31.476157 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.476135    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls" pod "6df63a7d-5b91-49f9-811b-ade411aaf0a4" (UID: "6df63a7d-5b91-49f9-811b-ade411aaf0a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:13:31.476245 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.476139    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r" pod "6df63a7d-5b91-49f9-811b-ade411aaf0a4" (UID: "6df63a7d-5b91-49f9-811b-ade411aaf0a4"). InnerVolumeSpecName "kube-api-access-ksm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:13:31.575237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.575160    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:13:31.575237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.575190    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:13:31.575237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.575201    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df63a7d-5b91-49f9-811b-ade411aaf0a4-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:13:31.575237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.575213    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:13:31.575237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:31.575222    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksm6r\" (UniqueName: \"kubernetes.io/projected/6df63a7d-5b91-49f9-811b-ade411aaf0a4-kube-api-access-ksm6r\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:13:32.126801 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.126761    4371 generic.go:356] "Generic (PLEG): container finished" podID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerID="840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df" exitCode=0
Oct 31 08:13:32.126981 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.126830    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"
Oct 31 08:13:32.126981 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.126846    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerDied","Data":"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"}
Oct 31 08:13:32.126981 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.126912    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m" event={"ID":"6df63a7d-5b91-49f9-811b-ade411aaf0a4","Type":"ContainerDied","Data":"84403eb261f56e698184c68d19912987c46611fffc78564629e5a699822ddfd8"}
Oct 31 08:13:32.126981 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.126938    4371 scope.go:122] "RemoveContainer" containerID="7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"
Oct 31 08:13:32.135363 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.135330    4371 scope.go:122] "RemoveContainer" containerID="840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"
Oct 31 08:13:32.142969 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.142951    4371 scope.go:122] "RemoveContainer" containerID="d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"
Oct 31 08:13:32.145466 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.145445    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:13:32.149334 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.149312    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-runtime-predictor-746b7f87bf-djj4m"]
Oct 31 08:13:32.151432 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.151417    4371 scope.go:122] "RemoveContainer" containerID="7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"
Oct 31 08:13:32.151738 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:13:32.151713    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802\": container with ID starting with 7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802 not found: ID does not exist" containerID="7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"
Oct 31 08:13:32.151816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.151746    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802"} err="failed to get container status \"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802\": rpc error: code = NotFound desc = could not find container \"7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802\": container with ID starting with 7a47d1c846e46c207462b02de7fabbdb3194cb6dc28bd5dc658e8fd6a53f2802 not found: ID does not exist"
Oct 31 08:13:32.151816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.151764    4371 scope.go:122] "RemoveContainer" containerID="840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"
Oct 31 08:13:32.152012 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:13:32.151998    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df\": container with ID starting with 840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df not found: ID does not exist" containerID="840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"
Oct 31 08:13:32.152060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.152017    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df"} err="failed to get container status \"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df\": rpc error: code = NotFound desc = could not find container \"840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df\": container with ID starting with 840262c5dcfbd0f08bc2ebb6658b5d572bbd6b9f08ee3cd36d3b48952cb401df not found: ID does not exist"
Oct 31 08:13:32.152060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.152029    4371 scope.go:122] "RemoveContainer" containerID="d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"
Oct 31 08:13:32.152251 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:13:32.152235    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934\": container with ID starting with d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934 not found: ID does not exist" containerID="d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"
Oct 31 08:13:32.152300 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:32.152254    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934"} err="failed to get container status \"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934\": rpc error: code = NotFound desc = could not find container \"d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934\": container with ID starting with d8751e101e36c282ca51293ee7dc067205e5425c77cef2ece65ee284c9bbe934 not found: ID does not exist"
Oct 31 08:13:33.214853 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:33.214800    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:33.331041 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:33.331005    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:33.704252 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:13:33.704220    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" path="/var/lib/kubelet/pods/6df63a7d-5b91-49f9-811b-ade411aaf0a4/volumes"
Oct 31 08:13:59.789466 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:13:59.789428    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:14:03.102804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:03.102768    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:22.434658 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:22.434620    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:14:22.435102 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:22.434941    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" containerID="cri-o://31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530" gracePeriod=30
Oct 31 08:14:22.435102 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:22.435001    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kube-rbac-proxy" containerID="cri-o://94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0" gracePeriod=30
Oct 31 08:14:22.495050 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:22.494986    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:23.317628 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:23.317595    4371 generic.go:356] "Generic (PLEG): container finished" podID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerID="94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0" exitCode=0
Oct 31 08:14:23.317822 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:23.317669    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerDied","Data":"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"}
Oct 31 08:14:24.585103 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:24.585058    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Oct 31 08:14:24.590257 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:24.590220    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 31 08:14:27.287436 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.287412    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:14:27.334360 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.334327    4371 generic.go:356] "Generic (PLEG): container finished" podID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerID="31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530" exitCode=0
Oct 31 08:14:27.334539 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.334401    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"
Oct 31 08:14:27.334539 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.334398    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerDied","Data":"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"}
Oct 31 08:14:27.334642 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.334545    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn" event={"ID":"218a9d59-74da-40a4-8cb5-d107bdd0d70b","Type":"ContainerDied","Data":"061cd8856bfe960212dbf470dd8ca9e9da7bc266d068658e92beab5b3c667b8d"}
Oct 31 08:14:27.334642 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.334572    4371 scope.go:122] "RemoveContainer" containerID="94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"
Oct 31 08:14:27.342961 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.342940    4371 scope.go:122] "RemoveContainer" containerID="31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"
Oct 31 08:14:27.351712 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.351636    4371 scope.go:122] "RemoveContainer" containerID="da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"
Oct 31 08:14:27.361303 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.361278    4371 scope.go:122] "RemoveContainer" containerID="94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"
Oct 31 08:14:27.361617 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:14:27.361594    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0\": container with ID starting with 94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0 not found: ID does not exist" containerID="94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"
Oct 31 08:14:27.361678 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.361632    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0"} err="failed to get container status \"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0\": rpc error: code = NotFound desc = could not find container \"94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0\": container with ID starting with 94e679b2f1cc804b64477c19d9662885a8cfa85ad871dd63fac48e432e6ed6e0 not found: ID does not exist"
Oct 31 08:14:27.361678 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.361654    4371 scope.go:122] "RemoveContainer" containerID="31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"
Oct 31 08:14:27.361916 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:14:27.361890    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530\": container with ID starting with 31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530 not found: ID does not exist" containerID="31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"
Oct 31 08:14:27.361986 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.361917    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530"} err="failed to get container status \"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530\": rpc error: code = NotFound desc = could not find container \"31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530\": container with ID starting with 31a575d70b260e9655fed8cf3f65783635838dc3a89ac036c1aea9c18e29d530 not found: ID does not exist"
Oct 31 08:14:27.361986 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.361932    4371 scope.go:122] "RemoveContainer" containerID="da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"
Oct 31 08:14:27.362190 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:14:27.362171    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef\": container with ID starting with da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef not found: ID does not exist" containerID="da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"
Oct 31 08:14:27.362234 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.362196    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef"} err="failed to get container status \"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef\": rpc error: code = NotFound desc = could not find container \"da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef\": container with ID starting with da4339445425b7fc70077da18c1029d158bc4d60f6ffa6ee408d2f93da8673ef not found: ID does not exist"
Oct 31 08:14:27.416723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.416626    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\") pod \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") "
Oct 31 08:14:27.416723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.416694    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\") pod \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") "
Oct 31 08:14:27.416979 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.416724    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\" (UniqueName: \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\") pod \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") "
Oct 31 08:14:27.416979 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.416768    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\") pod \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") "
Oct 31 08:14:27.416979 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.416803    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\") pod \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\" (UID: \"218a9d59-74da-40a4-8cb5-d107bdd0d70b\") "
Oct 31 08:14:27.417141 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.417035    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert" pod "218a9d59-74da-40a4-8cb5-d107bdd0d70b" (UID: "218a9d59-74da-40a4-8cb5-d107bdd0d70b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:27.417209 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.417154    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location" pod "218a9d59-74da-40a4-8cb5-d107bdd0d70b" (UID: "218a9d59-74da-40a4-8cb5-d107bdd0d70b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:27.417261 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.417242    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config" pod "218a9d59-74da-40a4-8cb5-d107bdd0d70b" (UID: "218a9d59-74da-40a4-8cb5-d107bdd0d70b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:27.418955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.418927    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls" pod "218a9d59-74da-40a4-8cb5-d107bdd0d70b" (UID: "218a9d59-74da-40a4-8cb5-d107bdd0d70b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:14:27.419066 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.419021    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8" pod "218a9d59-74da-40a4-8cb5-d107bdd0d70b" (UID: "218a9d59-74da-40a4-8cb5-d107bdd0d70b"). InnerVolumeSpecName "kube-api-access-99gf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:14:27.518008 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.517969    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:27.518008 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.518000    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/218a9d59-74da-40a4-8cb5-d107bdd0d70b-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:27.518008 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.518010    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99gf8\" (UniqueName: \"kubernetes.io/projected/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-api-access-99gf8\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:27.518008 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.518021    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:27.518296 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.518030    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/218a9d59-74da-40a4-8cb5-d107bdd0d70b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:27.657154 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.657120    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:14:27.662980 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.662958    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-8stkn"]
Oct 31 08:14:27.705046 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:27.704975    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" path="/var/lib/kubelet/pods/218a9d59-74da-40a4-8cb5-d107bdd0d70b/volumes"
Oct 31 08:14:28.995804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:28.995772    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:29.021368 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021336    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:14:29.021648 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021637    4371 cpu_manager.go:414] "containerMap: removing container" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kserve-container"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021649    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kserve-container"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021666    4371 cpu_manager.go:414] "containerMap: removing container" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021673    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021681    4371 cpu_manager.go:414] "containerMap: removing container" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021686    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021693    4371 cpu_manager.go:414] "containerMap: removing container" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="storage-initializer"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021698    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="storage-initializer"
Oct 31 08:14:29.021705 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021706    4371 cpu_manager.go:414] "containerMap: removing container" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021711    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021720    4371 cpu_manager.go:414] "containerMap: removing container" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021724    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021731    4371 cpu_manager.go:414] "containerMap: removing container" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021735    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021742    4371 cpu_manager.go:414] "containerMap: removing container" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="storage-initializer"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021747    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="storage-initializer"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021755    4371 cpu_manager.go:414] "containerMap: removing container" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="storage-initializer"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021760    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="storage-initializer"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021799    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kserve-container"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021809    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kserve-container"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021816    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kserve-container"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021821    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="218a9d59-74da-40a4-8cb5-d107bdd0d70b" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021827    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="b982ab2a-6a40-419a-84fd-8883a9a1ef17" containerName="kube-rbac-proxy"
Oct 31 08:14:29.021944 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.021832    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df63a7d-5b91-49f9-811b-ade411aaf0a4" containerName="kube-rbac-proxy"
Oct 31 08:14:29.026204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.026188    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.034237 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.034213    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:14:29.043769 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.043745    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:14:29.043909 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.043812    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 31 08:14:29.044045 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.044028    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:14:29.081190 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.081148    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 31 08:14:29.081382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.081315    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:14:29.081382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.081330    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:14:29.081473 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.081396    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:14:29.112606 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.112533    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:29.132632 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.132599    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.132762 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.132642    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.132762 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.132675    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887g5\" (UniqueName: \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.132942 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.132838    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.132942 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.132906    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.234251 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.234215    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887g5\" (UniqueName: \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.234415 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.234264    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.234415 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.234297    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.234415 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.234316    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.234415 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.234339    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.235240 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.235212    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.240734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.235555    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.240734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.235604    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.240734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.237825    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.258777 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.258714    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887g5\" (UniqueName: \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\") pod \"isvc-canary-stop-predictor-5485fd5dbd-5b4hj\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.290561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.290527    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:29.344672 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.344641    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:29.616719 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:29.616586    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:14:29.619338 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:14:29.619310    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc451f34_0403_4b97_a161_42b5e18685c8.slice/crio-2d4e94078525da1504ea1a113af0ec4b485efd8acfa26ec63eba83a5af18a5cf WatchSource:0}: Error finding container 2d4e94078525da1504ea1a113af0ec4b485efd8acfa26ec63eba83a5af18a5cf: Status 404 returned error can't find the container with id 2d4e94078525da1504ea1a113af0ec4b485efd8acfa26ec63eba83a5af18a5cf
Oct 31 08:14:29.789945 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:14:29.789906    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:14:30.348999 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:30.348963    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerStarted","Data":"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"}
Oct 31 08:14:30.349384 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:30.349025    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerStarted","Data":"2d4e94078525da1504ea1a113af0ec4b485efd8acfa26ec63eba83a5af18a5cf"}
Oct 31 08:14:34.365845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:34.365814    4371 generic.go:356] "Generic (PLEG): container finished" podID="dc451f34-0403-4b97-a161-42b5e18685c8" containerID="149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7" exitCode=0
Oct 31 08:14:34.365845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:34.365841    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerDied","Data":"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"}
Oct 31 08:14:35.371548 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:35.371489    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerStarted","Data":"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"}
Oct 31 08:14:35.371955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:35.371606    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerStarted","Data":"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"}
Oct 31 08:14:35.371955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:35.371811    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:35.393174 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:35.393125    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podStartSLOduration=2.237539142 podStartE2EDuration="6.393112092s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.15557295s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:14:35.390935098 +0000 UTC m=+2717.807065769" watchObservedRunningTime="2026-10-31 08:14:35.393112092 +0000 UTC m=+2717.809242764"
Oct 31 08:14:36.376708 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:36.376671    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:36.378093 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:36.378063    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:14:37.380528 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:37.380474    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:14:42.385404 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:42.385374    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:14:42.385928 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:42.385898    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:14:52.386164 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:52.386126    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:14:54.301974 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:14:54.301935    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:59.790058 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:14:59.790019    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:15:01.921061 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.920978    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:15:01.924358 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.924339    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:01.930826 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.930751    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"storage-config\""
Oct 31 08:15:01.933471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.933418    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 31 08:15:01.933658 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.933607    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:15:01.936501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.936474    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:15:01.936649 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.936594    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:15:01.936649 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.936641    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:15:01.936774 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.936651    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 31 08:15:01.940258 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.940233    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:01.997917 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.997885    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:01.998105 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.997951    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:01.998105 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.998039    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxcs\" (UniqueName: \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:01.998105 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:01.998095    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.042732 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.042702    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:02.098943 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.098906    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.098943 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.098946    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxcs\" (UniqueName: \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.099198 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.098985    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.099198 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.099008    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.099392 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.099369    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.099677 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.099645    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.101546 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.101524    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.106707 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.106685    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxcs\" (UniqueName: \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\") pod \"isvc-sklearn-s3-predictor-56dfd4cb7-5hr62\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.243111 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.243068    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:02.386244 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.386212    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:02.513816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:02.513794    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:15:02.516035 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:15:02.515999    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7731b068_13e8_4b20_b3e2_8a103af0d6e7.slice/crio-4d1997c5817ebc25fa9515f8a8ee4a2e91367763951ac7ea95557abb59468b78 WatchSource:0}: Error finding container 4d1997c5817ebc25fa9515f8a8ee4a2e91367763951ac7ea95557abb59468b78: Status 404 returned error can't find the container with id 4d1997c5817ebc25fa9515f8a8ee4a2e91367763951ac7ea95557abb59468b78
Oct 31 08:15:03.149860 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:03.149814    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:03.479699 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:03.479651    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerStarted","Data":"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"}
Oct 31 08:15:03.479897 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:03.479754    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerStarted","Data":"4d1997c5817ebc25fa9515f8a8ee4a2e91367763951ac7ea95557abb59468b78"}
Oct 31 08:15:04.485369 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:04.485334    4371 generic.go:356] "Generic (PLEG): container finished" podID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerID="adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a" exitCode=0
Oct 31 08:15:04.485786 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:04.485429    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerDied","Data":"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"}
Oct 31 08:15:05.491274 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.491240    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerStarted","Data":"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"}
Oct 31 08:15:05.491801 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.491302    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerStarted","Data":"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"}
Oct 31 08:15:05.491801 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.491455    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:05.491801 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.491494    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:05.492847 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.492818    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:05.510968 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:05.510925    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podStartSLOduration=3.021538481 podStartE2EDuration="4.510911685s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.489373204s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:15:05.510807352 +0000 UTC m=+2747.926938024" watchObservedRunningTime="2026-10-31 08:15:05.510911685 +0000 UTC m=+2747.927042357"
Oct 31 08:15:06.495979 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:06.495936    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:11.500753 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:11.500726    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:15:11.501388 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:11.501355    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:12.386270 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:12.386230    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:21.502092 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:21.502043    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:22.386205 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:22.386165    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:29.789377 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:15:29.789337    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:15:31.501362 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:31.501328    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:32.385837 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:32.385797    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:41.501225 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:41.501188    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:42.386275 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:42.386226    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:51.501683 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:51.501635    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:15:52.386801 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:52.386762    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 31 08:15:58.550495 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.550459    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:15:58.551070 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.550717    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container" containerID="cri-o://e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475" gracePeriod=30
Oct 31 08:15:58.551070 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.550755    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kube-rbac-proxy" containerID="cri-o://9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47" gracePeriod=30
Oct 31 08:15:58.551679 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.551655    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:15:58.700082 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.700051    4371 generic.go:356] "Generic (PLEG): container finished" podID="dc451f34-0403-4b97-a161-42b5e18685c8" containerID="9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47" exitCode=0
Oct 31 08:15:58.700256 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:15:58.700127    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerDied","Data":"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"}
Oct 31 08:15:59.789187 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:15:59.789150    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:16:01.503538 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:01.503478    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:16:02.381445 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:02.381403    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Oct 31 08:16:03.185609 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.185565    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:03.390309 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.390283    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:16:03.490710 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.490676    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\") pod \"dc451f34-0403-4b97-a161-42b5e18685c8\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") "
Oct 31 08:16:03.490880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.490755    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\") pod \"dc451f34-0403-4b97-a161-42b5e18685c8\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") "
Oct 31 08:16:03.490880 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.490840    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\") pod \"dc451f34-0403-4b97-a161-42b5e18685c8\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") "
Oct 31 08:16:03.490953 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.490895    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\") pod \"dc451f34-0403-4b97-a161-42b5e18685c8\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") "
Oct 31 08:16:03.490953 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.490925    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\" (UniqueName: \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\") pod \"dc451f34-0403-4b97-a161-42b5e18685c8\" (UID: \"dc451f34-0403-4b97-a161-42b5e18685c8\") "
Oct 31 08:16:03.491134 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.491022    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config" pod "dc451f34-0403-4b97-a161-42b5e18685c8" (UID: "dc451f34-0403-4b97-a161-42b5e18685c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:03.491196 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.491173    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert" pod "dc451f34-0403-4b97-a161-42b5e18685c8" (UID: "dc451f34-0403-4b97-a161-42b5e18685c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:03.491247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.491208    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location" pod "dc451f34-0403-4b97-a161-42b5e18685c8" (UID: "dc451f34-0403-4b97-a161-42b5e18685c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:16:03.492845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.492821    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls" pod "dc451f34-0403-4b97-a161-42b5e18685c8" (UID: "dc451f34-0403-4b97-a161-42b5e18685c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:16:03.492936 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.492908    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5" pod "dc451f34-0403-4b97-a161-42b5e18685c8" (UID: "dc451f34-0403-4b97-a161-42b5e18685c8"). InnerVolumeSpecName "kube-api-access-887g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:16:03.592444 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.592397    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887g5\" (UniqueName: \"kubernetes.io/projected/dc451f34-0403-4b97-a161-42b5e18685c8-kube-api-access-887g5\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:03.592444 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.592437    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:03.592444 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.592448    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc451f34-0403-4b97-a161-42b5e18685c8-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:03.592689 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.592459    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc451f34-0403-4b97-a161-42b5e18685c8-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:03.592689 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.592469    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc451f34-0403-4b97-a161-42b5e18685c8-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:03.718641 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.718613    4371 generic.go:356] "Generic (PLEG): container finished" podID="dc451f34-0403-4b97-a161-42b5e18685c8" containerID="e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475" exitCode=0
Oct 31 08:16:03.718788 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.718674    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"
Oct 31 08:16:03.718788 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.718699    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerDied","Data":"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"}
Oct 31 08:16:03.718788 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.718764    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj" event={"ID":"dc451f34-0403-4b97-a161-42b5e18685c8","Type":"ContainerDied","Data":"2d4e94078525da1504ea1a113af0ec4b485efd8acfa26ec63eba83a5af18a5cf"}
Oct 31 08:16:03.718896 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.718789    4371 scope.go:122] "RemoveContainer" containerID="9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"
Oct 31 08:16:03.727498 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.727477    4371 scope.go:122] "RemoveContainer" containerID="e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"
Oct 31 08:16:03.736871 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.736812    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:16:03.737617 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.737486    4371 scope.go:122] "RemoveContainer" containerID="149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"
Oct 31 08:16:03.740204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.740186    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-5b4hj"]
Oct 31 08:16:03.745709 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.745684    4371 scope.go:122] "RemoveContainer" containerID="9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"
Oct 31 08:16:03.745966 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:03.745947    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47\": container with ID starting with 9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47 not found: ID does not exist" containerID="9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"
Oct 31 08:16:03.746052 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.745980    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47"} err="failed to get container status \"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47\": rpc error: code = NotFound desc = could not find container \"9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47\": container with ID starting with 9a4e3e25d6b2ab92246809882061c3c40b60c4888a11ec56702c0539c95d4a47 not found: ID does not exist"
Oct 31 08:16:03.746052 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.746009    4371 scope.go:122] "RemoveContainer" containerID="e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"
Oct 31 08:16:03.746289 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:03.746272    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475\": container with ID starting with e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475 not found: ID does not exist" containerID="e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"
Oct 31 08:16:03.746332 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.746296    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475"} err="failed to get container status \"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475\": rpc error: code = NotFound desc = could not find container \"e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475\": container with ID starting with e177120c42c79416e35b8825c7059fb25ab7ee797b020fe8453be9e84cdbb475 not found: ID does not exist"
Oct 31 08:16:03.746332 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.746311    4371 scope.go:122] "RemoveContainer" containerID="149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"
Oct 31 08:16:03.746530 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:03.746496    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7\": container with ID starting with 149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7 not found: ID does not exist" containerID="149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"
Oct 31 08:16:03.746594 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:03.746540    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7"} err="failed to get container status \"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7\": rpc error: code = NotFound desc = could not find container \"149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7\": container with ID starting with 149a2de5179d18b97b9772f8bbe6dbcd7adcf7ddda4ed18755b5b10b73cb96b7 not found: ID does not exist"
Oct 31 08:16:05.703409 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:05.703371    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" path="/var/lib/kubelet/pods/dc451f34-0403-4b97-a161-42b5e18685c8/volumes"
Oct 31 08:16:09.178100 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:09.178053    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:09.278028 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:09.277982    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:11.502204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:11.502165    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:16:16.739223 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:16.739184    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:16.878499 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:16.878463    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:21.502144 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:21.502107    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 31 08:16:29.790478 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:29.790433    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:16:31.501693 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:31.501665    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:16:32.764301 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:32.764264    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:16:32.764829 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:32.764520    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container" containerID="cri-o://9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449" gracePeriod=30
Oct 31 08:16:32.764829 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:32.764566    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kube-rbac-proxy" containerID="cri-o://af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1" gracePeriod=30
Oct 31 08:16:32.832309 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:32.832262    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:33.841170 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:33.841136    4371 generic.go:356] "Generic (PLEG): container finished" podID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerID="af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1" exitCode=0
Oct 31 08:16:33.841581 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:33.841204    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerDied","Data":"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"}
Oct 31 08:16:36.496873 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:36.496833    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Oct 31 08:16:38.006521 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.006479    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:16:38.079450 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079415    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\" (UniqueName: \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\") pod \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") "
Oct 31 08:16:38.079673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079463    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\") pod \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") "
Oct 31 08:16:38.079673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079529    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\") pod \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") "
Oct 31 08:16:38.079673 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079585    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\") pod \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\" (UID: \"7731b068-13e8-4b20-b3e2-8a103af0d6e7\") "
Oct 31 08:16:38.079886 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079861    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config" pod "7731b068-13e8-4b20-b3e2-8a103af0d6e7" (UID: "7731b068-13e8-4b20-b3e2-8a103af0d6e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:38.079938 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.079903    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location" pod "7731b068-13e8-4b20-b3e2-8a103af0d6e7" (UID: "7731b068-13e8-4b20-b3e2-8a103af0d6e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:16:38.081632 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.081612    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs" pod "7731b068-13e8-4b20-b3e2-8a103af0d6e7" (UID: "7731b068-13e8-4b20-b3e2-8a103af0d6e7"). InnerVolumeSpecName "kube-api-access-qdxcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:16:38.081712 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.081662    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls" pod "7731b068-13e8-4b20-b3e2-8a103af0d6e7" (UID: "7731b068-13e8-4b20-b3e2-8a103af0d6e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:16:38.181038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.180999    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:38.181038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.181030    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxcs\" (UniqueName: \"kubernetes.io/projected/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-api-access-qdxcs\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:38.181038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.181041    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7731b068-13e8-4b20-b3e2-8a103af0d6e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:38.181038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.181051    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7731b068-13e8-4b20-b3e2-8a103af0d6e7-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:38.862655 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.862615    4371 generic.go:356] "Generic (PLEG): container finished" podID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerID="9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449" exitCode=0
Oct 31 08:16:38.862831 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.862675    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"
Oct 31 08:16:38.862831 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.862687    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerDied","Data":"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"}
Oct 31 08:16:38.862831 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.862739    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62" event={"ID":"7731b068-13e8-4b20-b3e2-8a103af0d6e7","Type":"ContainerDied","Data":"4d1997c5817ebc25fa9515f8a8ee4a2e91367763951ac7ea95557abb59468b78"}
Oct 31 08:16:38.862831 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.862768    4371 scope.go:122] "RemoveContainer" containerID="af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"
Oct 31 08:16:38.871254 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.871230    4371 scope.go:122] "RemoveContainer" containerID="9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"
Oct 31 08:16:38.878771 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.878755    4371 scope.go:122] "RemoveContainer" containerID="adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"
Oct 31 08:16:38.884948 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.884921    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:16:38.886348 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.886324    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-predictor-56dfd4cb7-5hr62"]
Oct 31 08:16:38.886866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.886717    4371 scope.go:122] "RemoveContainer" containerID="af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"
Oct 31 08:16:38.886998 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:38.886979    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1\": container with ID starting with af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1 not found: ID does not exist" containerID="af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"
Oct 31 08:16:38.887038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.887011    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1"} err="failed to get container status \"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1\": rpc error: code = NotFound desc = could not find container \"af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1\": container with ID starting with af970d733382860369457058a3555b6d24103b5aeab6ac56186b2816ea8fd0f1 not found: ID does not exist"
Oct 31 08:16:38.887038 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.887028    4371 scope.go:122] "RemoveContainer" containerID="9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"
Oct 31 08:16:38.887255 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:38.887237    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449\": container with ID starting with 9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449 not found: ID does not exist" containerID="9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"
Oct 31 08:16:38.887317 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.887266    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449"} err="failed to get container status \"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449\": rpc error: code = NotFound desc = could not find container \"9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449\": container with ID starting with 9d5d32860f949617a8b1005499b6203470f05cd0fcbd83ef5aa97978dc10b449 not found: ID does not exist"
Oct 31 08:16:38.887317 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.887289    4371 scope.go:122] "RemoveContainer" containerID="adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"
Oct 31 08:16:38.887582 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:38.887494    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a\": container with ID starting with adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a not found: ID does not exist" containerID="adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"
Oct 31 08:16:38.887628 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:38.887588    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a"} err="failed to get container status \"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a\": rpc error: code = NotFound desc = could not find container \"adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a\": container with ID starting with adf3c60e776945a62f3fe96c6b90a8f2f89560287ec39cd68f6e085b7276cc9a not found: ID does not exist"
Oct 31 08:16:39.703579 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:39.703547    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" path="/var/lib/kubelet/pods/7731b068-13e8-4b20-b3e2-8a103af0d6e7/volumes"
Oct 31 08:16:40.316611 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:40.316554    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:40.431375 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:16:40.431337    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:59.789805 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:16:59.789768    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:17:03.262742 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:17:03.262703    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:29.789568 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:17:29.789505    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:17:59.789628 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:17:59.789593    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:18:03.271727 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:03.271642    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:07.809185 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:07.809153    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:15.207835 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:15.207795    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:15.417757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:15.417722    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:20.494035 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:20.493985    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:21.631550 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631501    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631834    4371 cpu_manager.go:414] "containerMap: removing container" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631847    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631862    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="storage-initializer"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631868    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="storage-initializer"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631875    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kube-rbac-proxy"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631880    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kube-rbac-proxy"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631896    4371 cpu_manager.go:414] "containerMap: removing container" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kube-rbac-proxy"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631900    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kube-rbac-proxy"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631907    4371 cpu_manager.go:414] "containerMap: removing container" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631913    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631933    4371 cpu_manager.go:414] "containerMap: removing container" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="storage-initializer"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631939    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="storage-initializer"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631985    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kube-rbac-proxy"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.631992    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.632003    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc451f34-0403-4b97-a161-42b5e18685c8" containerName="kserve-container"
Oct 31 08:18:21.632031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.632013    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="7731b068-13e8-4b20-b3e2-8a103af0d6e7" containerName="kube-rbac-proxy"
Oct 31 08:18:21.634982 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.634966    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.645035 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.644903    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Oct 31 08:18:21.645035 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.644991    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Oct 31 08:18:21.645711 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.645692    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:18:21.645953 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.645938    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"storage-config\""
Oct 31 08:18:21.646040 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.645972    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:18:21.646040 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.646013    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:18:21.646142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.646053    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:21.651537 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.651497    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:21.800285 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.800246    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.800471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.800302    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p266m\" (UniqueName: \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.800471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.800380    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.800471 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.800422    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.827205 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.827177    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:21.901618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.901496    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.901618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.901584    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p266m\" (UniqueName: \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.901618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.901611    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.901924 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.901639    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.902051 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.902023    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.902316 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.902291    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.904031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.904012    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.909861 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.909835    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p266m\" (UniqueName: \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:21.952654 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:21.952613    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:22.224712 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:22.224687    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:22.227101 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:18:22.227071    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b23d8a_d705_4bdd_84e0_9e924ea1e7ac.slice/crio-024818695c968528cb52e963c66ee44d56c11f6536dafaed685071ee3ddbd158 WatchSource:0}: Error finding container 024818695c968528cb52e963c66ee44d56c11f6536dafaed685071ee3ddbd158: Status 404 returned error can't find the container with id 024818695c968528cb52e963c66ee44d56c11f6536dafaed685071ee3ddbd158
Oct 31 08:18:23.234780 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:23.234745    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerStarted","Data":"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"}
Oct 31 08:18:23.235178 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:23.234804    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerStarted","Data":"024818695c968528cb52e963c66ee44d56c11f6536dafaed685071ee3ddbd158"}
Oct 31 08:18:27.251702 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:27.251675    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/0.log"
Oct 31 08:18:27.252055 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:27.251713    4371 generic.go:356] "Generic (PLEG): container finished" podID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerID="c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3" exitCode=1
Oct 31 08:18:27.252055 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:27.251759    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerDied","Data":"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"}
Oct 31 08:18:28.257557 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:28.257501    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/0.log"
Oct 31 08:18:28.257968 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:28.257616    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerStarted","Data":"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"}
Oct 31 08:18:29.789437 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:18:29.789402    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:18:31.628373 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.628334    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:31.628876 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.628676    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer" containerID="cri-o://6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469" gracePeriod=30
Oct 31 08:18:31.696127 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.696076    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:31.863915 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.863891    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/1.log"
Oct 31 08:18:31.864304 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.864288    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/0.log"
Oct 31 08:18:31.866547 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.866529    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:31.986711 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.986681    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\") pod \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") "
Oct 31 08:18:31.986899 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.986724    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\" (UniqueName: \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\") pod \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") "
Oct 31 08:18:31.986899 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.986780    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\") pod \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") "
Oct 31 08:18:31.986899 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.986809    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\") pod \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\" (UID: \"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac\") "
Oct 31 08:18:31.987155 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.987124    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location" pod "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" (UID: "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:31.987272 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.987146    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config" pod "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" (UID: "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:18:31.988884 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.988858    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m" pod "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" (UID: "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac"). InnerVolumeSpecName "kube-api-access-p266m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:18:31.988960 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:31.988888    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls" pod "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" (UID: "20b23d8a-d705-4bdd-84e0-9e924ea1e7ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:18:32.088150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.088092    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:32.088150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.088143    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p266m\" (UniqueName: \"kubernetes.io/projected/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-api-access-p266m\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:32.088150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.088154    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:32.088150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.088165    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:32.274661 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.274580    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/1.log"
Oct 31 08:18:32.274998 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.274981    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27_20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/storage-initializer/0.log"
Oct 31 08:18:32.275106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.275021    4371 generic.go:356] "Generic (PLEG): container finished" podID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerID="6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469" exitCode=1
Oct 31 08:18:32.275106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.275084    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"
Oct 31 08:18:32.275206 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.275081    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerDied","Data":"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"}
Oct 31 08:18:32.275259 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.275226    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27" event={"ID":"20b23d8a-d705-4bdd-84e0-9e924ea1e7ac","Type":"ContainerDied","Data":"024818695c968528cb52e963c66ee44d56c11f6536dafaed685071ee3ddbd158"}
Oct 31 08:18:32.275314 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.275269    4371 scope.go:122] "RemoveContainer" containerID="6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"
Oct 31 08:18:32.284248 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.284225    4371 scope.go:122] "RemoveContainer" containerID="c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"
Oct 31 08:18:32.291723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.291699    4371 scope.go:122] "RemoveContainer" containerID="6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"
Oct 31 08:18:32.291988 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:18:32.291968    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469\": container with ID starting with 6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469 not found: ID does not exist" containerID="6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"
Oct 31 08:18:32.292029 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.292000    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469"} err="failed to get container status \"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469\": rpc error: code = NotFound desc = could not find container \"6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469\": container with ID starting with 6c5955831da53f7bda4ec118a892e1819472f9eafda8fed0c40a4f7e8d129469 not found: ID does not exist"
Oct 31 08:18:32.292029 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.292015    4371 scope.go:122] "RemoveContainer" containerID="c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"
Oct 31 08:18:32.292262 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:18:32.292248    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3\": container with ID starting with c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3 not found: ID does not exist" containerID="c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"
Oct 31 08:18:32.292298 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.292269    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3"} err="failed to get container status \"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3\": rpc error: code = NotFound desc = could not find container \"c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3\": container with ID starting with c44e1baf972a3801eb2834e0c0c9282c951a7813f362b51394f5f93308dafec3 not found: ID does not exist"
Oct 31 08:18:32.310089 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.310063    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:32.313987 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:32.313964    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-fail-predictor-667f56c98c-9mb27"]
Oct 31 08:18:33.071314 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:33.071273    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:33.177430 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:33.177400    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:33.703300 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:18:33.703269    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" path="/var/lib/kubelet/pods/20b23d8a-d705-4bdd-84e0-9e924ea1e7ac/volumes"
Oct 31 08:18:59.789431 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:18:59.789390    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:19:03.324679 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:19:03.324642    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:29.790897 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:19:29.790855    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:19:59.789534 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:19:59.789436    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:20:03.369098 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:03.369060    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:06.324950 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:06.324910    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:06.640243 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:06.640168    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:13.230666 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:13.230629    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:14.022359 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.022327    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:14.046573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.046546    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:20:14.046855 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.046841    4371 cpu_manager.go:414] "containerMap: removing container" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.046897 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.046858    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.046943 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.046933    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.047001 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.046991    4371 cpu_manager.go:414] "containerMap: removing container" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.047040 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.047002    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.047079 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.047070    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b23d8a-d705-4bdd-84e0-9e924ea1e7ac" containerName="storage-initializer"
Oct 31 08:20:14.050059 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.050034    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.058991 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.058956    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 31 08:20:14.059126 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.059006    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:20:14.059126 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.059071    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:20:14.065106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.059537    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:20:14.065106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.059615    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:20:14.065106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.060040    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:20:14.065106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.061234    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:20:14.065106 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.063482    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:20:14.132241 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.132202    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:14.203972 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.203928    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqqb\" (UniqueName: \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.204136 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.203987    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.204136 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.204074    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.204136 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.204124    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.204247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.204228    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.305472 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305389    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.305472 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305438    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.305472 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305471    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.306042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305497    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.306042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305577    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqqb\" (UniqueName: \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.306042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.305920    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.306154 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.306121    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.306201 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.306187    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.307976 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.307955    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.313649 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.313625    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqqb\" (UniqueName: \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\") pod \"isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.374744 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.374715    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:14.426202 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.426005    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:14.523642 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.523614    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:14.646357 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.646151    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:20:14.649038 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:20:14.649003    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a893db_3aa8_4e33_9cf8_19379346dfed.slice/crio-744b75b978cdd9a579158b5b34a55866fa1ffafd275e793a93c024bda54080c1 WatchSource:0}: Error finding container 744b75b978cdd9a579158b5b34a55866fa1ffafd275e793a93c024bda54080c1: Status 404 returned error can't find the container with id 744b75b978cdd9a579158b5b34a55866fa1ffafd275e793a93c024bda54080c1
Oct 31 08:20:14.655277 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:14.655252    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerStarted","Data":"744b75b978cdd9a579158b5b34a55866fa1ffafd275e793a93c024bda54080c1"}
Oct 31 08:20:15.661373 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:15.661333    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerStarted","Data":"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"}
Oct 31 08:20:19.676790 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:19.676755    4371 generic.go:356] "Generic (PLEG): container finished" podID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerID="40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35" exitCode=0
Oct 31 08:20:19.677213 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:19.676815    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerDied","Data":"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"}
Oct 31 08:20:20.683381 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:20.683347    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerStarted","Data":"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"}
Oct 31 08:20:20.683850 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:20.683405    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerStarted","Data":"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"}
Oct 31 08:20:20.683850 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:20.683654    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:20.683850 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:20.683786    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:24.427117 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:24.427082    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:25.779436 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:25.779400    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:25.879152 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:25.879118    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:26.692754 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:26.692718    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:26.713818 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:26.713766    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" podStartSLOduration=8.478446318 podStartE2EDuration="12.713753959s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.235307641s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:20:20.704860066 +0000 UTC m=+3063.120990738" watchObservedRunningTime="2026-10-31 08:20:26.713753959 +0000 UTC m=+3069.129884631"
Oct 31 08:20:29.789734 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:20:29.789702    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:20:56.697216 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:20:56.697185    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:20:59.789940 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:20:59.789889    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:21:03.406779 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:03.406694    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:04.366840 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.366805    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:21:04.367078 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.367042    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kserve-container" containerID="cri-o://9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27" gracePeriod=30
Oct 31 08:21:04.367176 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.367111    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kube-rbac-proxy" containerID="cri-o://a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711" gracePeriod=30
Oct 31 08:21:04.430971 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.430931    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:04.848376 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.848343    4371 generic.go:356] "Generic (PLEG): container finished" podID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerID="a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711" exitCode=0
Oct 31 08:21:04.848573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:04.848417    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerDied","Data":"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"}
Oct 31 08:21:05.408928 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.408903    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:21:05.525600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525494    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\") pod \"a3a893db-3aa8-4e33-9cf8-19379346dfed\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") "
Oct 31 08:21:05.525600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525566    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\" (UniqueName: \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\") pod \"a3a893db-3aa8-4e33-9cf8-19379346dfed\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") "
Oct 31 08:21:05.525600 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525594    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\") pod \"a3a893db-3aa8-4e33-9cf8-19379346dfed\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") "
Oct 31 08:21:05.526166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525634    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\") pod \"a3a893db-3aa8-4e33-9cf8-19379346dfed\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") "
Oct 31 08:21:05.526166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525668    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\") pod \"a3a893db-3aa8-4e33-9cf8-19379346dfed\" (UID: \"a3a893db-3aa8-4e33-9cf8-19379346dfed\") "
Oct 31 08:21:05.526166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.525939    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert" pod "a3a893db-3aa8-4e33-9cf8-19379346dfed" (UID: "a3a893db-3aa8-4e33-9cf8-19379346dfed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:21:05.526166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.526054    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location" pod "a3a893db-3aa8-4e33-9cf8-19379346dfed" (UID: "a3a893db-3aa8-4e33-9cf8-19379346dfed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:05.526166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.526144    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config" pod "a3a893db-3aa8-4e33-9cf8-19379346dfed" (UID: "a3a893db-3aa8-4e33-9cf8-19379346dfed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:21:05.527646 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.527617    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls" pod "a3a893db-3aa8-4e33-9cf8-19379346dfed" (UID: "a3a893db-3aa8-4e33-9cf8-19379346dfed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:21:05.527757 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.527714    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb" pod "a3a893db-3aa8-4e33-9cf8-19379346dfed" (UID: "a3a893db-3aa8-4e33-9cf8-19379346dfed"). InnerVolumeSpecName "kube-api-access-qhqqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:21:05.626836 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.626799    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:05.626836 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.626829    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a893db-3aa8-4e33-9cf8-19379346dfed-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:05.626836 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.626839    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqqb\" (UniqueName: \"kubernetes.io/projected/a3a893db-3aa8-4e33-9cf8-19379346dfed-kube-api-access-qhqqb\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:05.627054 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.626847    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a893db-3aa8-4e33-9cf8-19379346dfed-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:05.627054 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.626858    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a893db-3aa8-4e33-9cf8-19379346dfed-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:05.853650 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.853563    4371 generic.go:356] "Generic (PLEG): container finished" podID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerID="9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27" exitCode=0
Oct 31 08:21:05.853650 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.853624    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"
Oct 31 08:21:05.853877 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.853620    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerDied","Data":"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"}
Oct 31 08:21:05.853877 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.853752    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4" event={"ID":"a3a893db-3aa8-4e33-9cf8-19379346dfed","Type":"ContainerDied","Data":"744b75b978cdd9a579158b5b34a55866fa1ffafd275e793a93c024bda54080c1"}
Oct 31 08:21:05.853877 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.853780    4371 scope.go:122] "RemoveContainer" containerID="a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"
Oct 31 08:21:05.862294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.862274    4371 scope.go:122] "RemoveContainer" containerID="9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"
Oct 31 08:21:05.869665 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.869636    4371 scope.go:122] "RemoveContainer" containerID="40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"
Oct 31 08:21:05.871846 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.871828    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:21:05.877044 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.877020    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-runtime-predictor-5496bf5548-95dp4"]
Oct 31 08:21:05.878001 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.877985    4371 scope.go:122] "RemoveContainer" containerID="a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"
Oct 31 08:21:05.878283 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:21:05.878265    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711\": container with ID starting with a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711 not found: ID does not exist" containerID="a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"
Oct 31 08:21:05.878330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.878293    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711"} err="failed to get container status \"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711\": rpc error: code = NotFound desc = could not find container \"a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711\": container with ID starting with a0b5ae6c00cd1435ae79b2ccfe03dbfe1267bde4d0e95379603cfd7dfae41711 not found: ID does not exist"
Oct 31 08:21:05.878330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.878310    4371 scope.go:122] "RemoveContainer" containerID="9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"
Oct 31 08:21:05.878548 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:21:05.878530    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27\": container with ID starting with 9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27 not found: ID does not exist" containerID="9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"
Oct 31 08:21:05.878622 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.878558    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27"} err="failed to get container status \"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27\": rpc error: code = NotFound desc = could not find container \"9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27\": container with ID starting with 9d7e915bf86736c513cbe40c7d8795b11551ce3ee2bd71ca7fc1028ff8a5ee27 not found: ID does not exist"
Oct 31 08:21:05.878622 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.878578    4371 scope.go:122] "RemoveContainer" containerID="40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"
Oct 31 08:21:05.878824 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:21:05.878803    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35\": container with ID starting with 40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35 not found: ID does not exist" containerID="40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"
Oct 31 08:21:05.878862 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:05.878828    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35"} err="failed to get container status \"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35\": rpc error: code = NotFound desc = could not find container \"40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35\": container with ID starting with 40dc81183c17bde1ed2deb93ec22cd2ab6e3ed08da6707197083a5b504108a35 not found: ID does not exist"
Oct 31 08:21:07.703482 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:07.703449    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" path="/var/lib/kubelet/pods/a3a893db-3aa8-4e33-9cf8-19379346dfed/volumes"
Oct 31 08:21:07.706606 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:07.706582    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:07.892412 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:07.892360    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:29.789751 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:21:29.789714    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:21:38.131046 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:38.131004    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:41.443347 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.443313    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:41.446260 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446234    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:21:41.446537 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446524    4371 cpu_manager.go:414] "containerMap: removing container" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kserve-container"
Oct 31 08:21:41.446585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446538    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kserve-container"
Oct 31 08:21:41.446585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446570    4371 cpu_manager.go:414] "containerMap: removing container" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kube-rbac-proxy"
Oct 31 08:21:41.446585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446575    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kube-rbac-proxy"
Oct 31 08:21:41.446585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446584    4371 cpu_manager.go:414] "containerMap: removing container" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="storage-initializer"
Oct 31 08:21:41.446713 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446589    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="storage-initializer"
Oct 31 08:21:41.446713 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446640    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kube-rbac-proxy"
Oct 31 08:21:41.446713 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.446649    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a893db-3aa8-4e33-9cf8-19379346dfed" containerName="kserve-container"
Oct 31 08:21:41.449623 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.449607    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.453438 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.453403    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:21:41.453438 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.453403    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:21:41.454473 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.454457    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:21:41.454592 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.454576    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:21:41.454660 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.454608    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:21:41.454660 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.454607    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:21:41.454756 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.454693    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 31 08:21:41.459630 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.459608    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:21:41.522262 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.522225    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.522262 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.522268    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhcf\" (UniqueName: \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.522470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.522298    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.522470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.522397    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.522470 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.522430    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.555999 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.555964    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:41.623182 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623142    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.623371 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623188    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.623371 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623277    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.623371 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623320    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhcf\" (UniqueName: \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.623371 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623365    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.623896 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.623870    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.624031 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.624010    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.624079 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.624021    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.625652 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.625633    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.631334 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.631311    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhcf\" (UniqueName: \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\") pod \"isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") " pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:41.767236 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:41.767206    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:42.047265 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:42.044725    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:21:42.049501 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:21:42.049460    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cb7437_712a_4ea4_83ae_0a09fb28edba.slice/crio-e47f4f424aa79bafe43285cf6c6a02f1909f8eba6b7a975a6a454cb56b98eecd WatchSource:0}: Error finding container e47f4f424aa79bafe43285cf6c6a02f1909f8eba6b7a975a6a454cb56b98eecd: Status 404 returned error can't find the container with id e47f4f424aa79bafe43285cf6c6a02f1909f8eba6b7a975a6a454cb56b98eecd
Oct 31 08:21:42.993551 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:42.993500    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerStarted","Data":"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"}
Oct 31 08:21:42.994000 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:42.993572    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerStarted","Data":"e47f4f424aa79bafe43285cf6c6a02f1909f8eba6b7a975a6a454cb56b98eecd"}
Oct 31 08:21:47.009669 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:47.009636    4371 generic.go:356] "Generic (PLEG): container finished" podID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerID="89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756" exitCode=0
Oct 31 08:21:47.010050 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:47.009713    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerDied","Data":"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"}
Oct 31 08:21:48.015690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:48.015656    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerStarted","Data":"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"}
Oct 31 08:21:48.016115 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:48.015720    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerStarted","Data":"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"}
Oct 31 08:21:48.016115 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:48.015932    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:48.042811 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:48.042760    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" podStartSLOduration=2.439500959 podStartE2EDuration="7.042746918s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.603245959s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:21:48.040394708 +0000 UTC m=+3150.456525380" watchObservedRunningTime="2026-10-31 08:21:48.042746918 +0000 UTC m=+3150.458877593"
Oct 31 08:21:49.019866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:49.019836    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:55.028832 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:21:55.028801    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:21:59.789044 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:21:59.789012    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:22:03.450651 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:03.450618    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:05.937208 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:05.937180    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:07.073908 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:07.073863    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:07.183773 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:07.183735    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:18.452292 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:18.452241    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:20.520223 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:20.520186    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:20.628355 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:20.628324    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:25.032960 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:25.032930    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:22:27.069706 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:27.069671    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:29.788797 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:22:29.788760    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:22:31.746395 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:31.746322    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:22:31.746844 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:31.746604    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kserve-container" containerID="cri-o://5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed" gracePeriod=30
Oct 31 08:22:31.746844 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:31.746633    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kube-rbac-proxy" containerID="cri-o://9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f" gracePeriod=30
Oct 31 08:22:31.797647 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:31.797618    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:32.175814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.175743    4371 generic.go:356] "Generic (PLEG): container finished" podID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerID="9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f" exitCode=0
Oct 31 08:22:32.175814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.175789    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerDied","Data":"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"}
Oct 31 08:22:32.883823 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.883794    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:22:32.928921 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.928866    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\") pod \"22cb7437-712a-4ea4-83ae-0a09fb28edba\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") "
Oct 31 08:22:32.928921 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.928898    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\" (UniqueName: \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\") pod \"22cb7437-712a-4ea4-83ae-0a09fb28edba\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") "
Oct 31 08:22:32.928921 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.928923    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\") pod \"22cb7437-712a-4ea4-83ae-0a09fb28edba\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") "
Oct 31 08:22:32.929119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.928962    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\" (UniqueName: \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\") pod \"22cb7437-712a-4ea4-83ae-0a09fb28edba\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") "
Oct 31 08:22:32.929119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.928994    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\") pod \"22cb7437-712a-4ea4-83ae-0a09fb28edba\" (UID: \"22cb7437-712a-4ea4-83ae-0a09fb28edba\") "
Oct 31 08:22:32.929278 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.929251    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location" pod "22cb7437-712a-4ea4-83ae-0a09fb28edba" (UID: "22cb7437-712a-4ea4-83ae-0a09fb28edba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:22:32.929368 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.929345    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert" pod "22cb7437-712a-4ea4-83ae-0a09fb28edba" (UID: "22cb7437-712a-4ea4-83ae-0a09fb28edba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:22:32.929454 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.929435    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config" pod "22cb7437-712a-4ea4-83ae-0a09fb28edba" (UID: "22cb7437-712a-4ea4-83ae-0a09fb28edba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:22:32.930964 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.930948    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf" pod "22cb7437-712a-4ea4-83ae-0a09fb28edba" (UID: "22cb7437-712a-4ea4-83ae-0a09fb28edba"). InnerVolumeSpecName "kube-api-access-zbhcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:22:32.931072 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:32.931054    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls" pod "22cb7437-712a-4ea4-83ae-0a09fb28edba" (UID: "22cb7437-712a-4ea4-83ae-0a09fb28edba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:22:33.030382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.030357    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22cb7437-712a-4ea4-83ae-0a09fb28edba-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:33.030382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.030381    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhcf\" (UniqueName: \"kubernetes.io/projected/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-api-access-zbhcf\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:33.030567 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.030392    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:33.030567 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.030401    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22cb7437-712a-4ea4-83ae-0a09fb28edba-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:33.030567 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.030410    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22cb7437-712a-4ea4-83ae-0a09fb28edba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:33.181902 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.181829    4371 generic.go:356] "Generic (PLEG): container finished" podID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerID="5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed" exitCode=0
Oct 31 08:22:33.181902 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.181889    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"
Oct 31 08:22:33.182081 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.181910    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerDied","Data":"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"}
Oct 31 08:22:33.182081 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.181971    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf" event={"ID":"22cb7437-712a-4ea4-83ae-0a09fb28edba","Type":"ContainerDied","Data":"e47f4f424aa79bafe43285cf6c6a02f1909f8eba6b7a975a6a454cb56b98eecd"}
Oct 31 08:22:33.182081 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.181996    4371 scope.go:122] "RemoveContainer" containerID="9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"
Oct 31 08:22:33.190346 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.190331    4371 scope.go:122] "RemoveContainer" containerID="5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"
Oct 31 08:22:33.197627 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.197609    4371 scope.go:122] "RemoveContainer" containerID="89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"
Oct 31 08:22:33.204787 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.204764    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:22:33.205099 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.205080    4371 scope.go:122] "RemoveContainer" containerID="9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"
Oct 31 08:22:33.205382 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:22:33.205360    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f\": container with ID starting with 9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f not found: ID does not exist" containerID="9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"
Oct 31 08:22:33.205459 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.205391    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f"} err="failed to get container status \"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f\": rpc error: code = NotFound desc = could not find container \"9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f\": container with ID starting with 9a2bb3243f2c95ee235ab665bb75f8fe57a31f17b26a73110a141278ceba890f not found: ID does not exist"
Oct 31 08:22:33.205459 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.205409    4371 scope.go:122] "RemoveContainer" containerID="5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"
Oct 31 08:22:33.205814 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:22:33.205760    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed\": container with ID starting with 5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed not found: ID does not exist" containerID="5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"
Oct 31 08:22:33.205975 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.205949    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed"} err="failed to get container status \"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed\": rpc error: code = NotFound desc = could not find container \"5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed\": container with ID starting with 5f8f870f68e6235683688b6add003be9ee36355395e3bcef68723b2f8d5149ed not found: ID does not exist"
Oct 31 08:22:33.205975 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.205973    4371 scope.go:122] "RemoveContainer" containerID="89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"
Oct 31 08:22:33.206266 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:22:33.206249    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756\": container with ID starting with 89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756 not found: ID does not exist" containerID="89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"
Oct 31 08:22:33.206331 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.206270    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756"} err="failed to get container status \"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756\": rpc error: code = NotFound desc = could not find container \"89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756\": container with ID starting with 89bc72b0f86a3ad86ca8b04e63f829c6ce27ed674268325a7a3a493e7f2e4756 not found: ID does not exist"
Oct 31 08:22:33.207520 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.207490    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-mlflow-v2-runtime-predictor-5f96d5685c-cljxf"]
Oct 31 08:22:33.703284 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:33.703250    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" path="/var/lib/kubelet/pods/22cb7437-712a-4ea4-83ae-0a09fb28edba/volumes"
Oct 31 08:22:35.120072 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.120039    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:35.126526 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126490    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:22:35.126814 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126800    4371 cpu_manager.go:414] "containerMap: removing container" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kserve-container"
Oct 31 08:22:35.126865 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126815    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kserve-container"
Oct 31 08:22:35.126865 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126823    4371 cpu_manager.go:414] "containerMap: removing container" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="storage-initializer"
Oct 31 08:22:35.126865 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126828    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="storage-initializer"
Oct 31 08:22:35.126865 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126837    4371 cpu_manager.go:414] "containerMap: removing container" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kube-rbac-proxy"
Oct 31 08:22:35.126865 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126842    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kube-rbac-proxy"
Oct 31 08:22:35.127033 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126890    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kserve-container"
Oct 31 08:22:35.127033 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.126898    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cb7437-712a-4ea4-83ae-0a09fb28edba" containerName="kube-rbac-proxy"
Oct 31 08:22:35.131505 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.131489    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.138860 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.138839    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Oct 31 08:22:35.138860 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.138852    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:22:35.140617 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.140597    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Oct 31 08:22:35.140783 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.140614    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:22:35.140867 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.140625    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:22:35.140867 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.140701    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:22:35.140867 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.140772    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:22:35.142262 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.142242    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:22:35.146619 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.146598    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.146717 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.146631    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.146791 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.146727    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffrq\" (UniqueName: \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.146791 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.146753    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.146898 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.146792    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247211    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffrq\" (UniqueName: \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247251    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247287    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247341    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247361    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.247738 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.247715    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.248083 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.248058    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.248166 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.248069    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.249723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.249705    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.255157 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.255134    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffrq\" (UniqueName: \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\") pod \"isvc-sklearn-mcp-predictor-698cffff67-gktj4\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") " pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.264282 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.264257    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:35.452578 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.452494    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:35.724350 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:35.724326    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:22:35.726392 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:22:35.726367    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b7dc11_8605_41c0_b705_26ef3c38f885.slice/crio-f0409b7a6f0886c9f7c7a2795fbdf90715e158d991c546c171dcab8d509fab27 WatchSource:0}: Error finding container f0409b7a6f0886c9f7c7a2795fbdf90715e158d991c546c171dcab8d509fab27: Status 404 returned error can't find the container with id f0409b7a6f0886c9f7c7a2795fbdf90715e158d991c546c171dcab8d509fab27
Oct 31 08:22:36.196936 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:36.196856    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerStarted","Data":"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"}
Oct 31 08:22:36.197386 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:36.196937    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerStarted","Data":"f0409b7a6f0886c9f7c7a2795fbdf90715e158d991c546c171dcab8d509fab27"}
Oct 31 08:22:40.215028 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:40.214944    4371 generic.go:356] "Generic (PLEG): container finished" podID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerID="f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617" exitCode=0
Oct 31 08:22:40.215390 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:40.215017    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerDied","Data":"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"}
Oct 31 08:22:40.277336 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:40.277320    4371 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:22:41.222887 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:41.222854    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerStarted","Data":"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"}
Oct 31 08:22:43.235745 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:43.235722    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerStarted","Data":"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"}
Oct 31 08:22:44.242077 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:44.242039    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerStarted","Data":"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"}
Oct 31 08:22:44.242581 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:44.242215    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:44.264667 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:44.264625    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podStartSLOduration=2.453944393 podStartE2EDuration="9.264613183s" totalImagesPullingTime="2.853457069s" totalInitContainerRuntime="3.957211721s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:22:44.262948994 +0000 UTC m=+3206.679079689" watchObservedRunningTime="2026-10-31 08:22:44.264613183 +0000 UTC m=+3206.680743855"
Oct 31 08:22:45.246576 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:45.246540    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:45.246959 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:45.246616    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:51.255496 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:22:51.255470    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:22:59.789498 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:22:59.789456    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:23:03.492279 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:03.492245    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:11.256879 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:11.256842    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 31 08:23:15.453769 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:15.453718    4371 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 31 08:23:21.439974 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:21.439938    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:23:29.789116 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:23:29.789077    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:23:32.401379 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:32.401351    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:37.702790 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:37.702754    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:37.944108 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:37.944075    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:51.440989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:51.440959    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:23:55.339815 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.339786    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:23:55.340304 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.340109    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" containerID="cri-o://74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14" gracePeriod=30
Oct 31 08:23:55.340304 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.340135    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-agent" containerID="cri-o://676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856" gracePeriod=30
Oct 31 08:23:55.340434 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.340299    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" containerID="cri-o://dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2" gracePeriod=30
Oct 31 08:23:55.406896 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.406867    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:55.501803 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.501773    4371 generic.go:356] "Generic (PLEG): container finished" podID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerID="74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14" exitCode=0
Oct 31 08:23:55.501952 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:55.501847    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerDied","Data":"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"}
Oct 31 08:23:56.251486 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:56.251445    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:23:58.515216 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:58.515182    4371 generic.go:356] "Generic (PLEG): container finished" podID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerID="dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2" exitCode=0
Oct 31 08:23:58.515630 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:23:58.515238    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerDied","Data":"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"}
Oct 31 08:23:59.788965 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:23:59.788932    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:24:01.251119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:01.251082    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:24:01.256446 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:01.256426    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 31 08:24:03.525366 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:03.525331    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:04.974370 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:04.974337    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:05.086058 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:05.086027    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:06.251432 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:06.251388    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:24:06.251839 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:06.251529    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:24:10.901264 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:10.901231    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:11.251125 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:11.251082    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:24:11.256484 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:11.256457    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 31 08:24:16.250776 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:16.250734    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:24:21.251154 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:21.251104    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 31 08:24:21.256551 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:21.256498    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 31 08:24:21.256648 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:21.256635    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:24:25.529176 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.529154    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:24:25.610046 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.609962    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\") pod \"17b7dc11-8605-41c0-b705-26ef3c38f885\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") "
Oct 31 08:24:25.610046 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610019    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\") pod \"17b7dc11-8605-41c0-b705-26ef3c38f885\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") "
Oct 31 08:24:25.610294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610059    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\") pod \"17b7dc11-8605-41c0-b705-26ef3c38f885\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") "
Oct 31 08:24:25.610294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610125    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\" (UniqueName: \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\") pod \"17b7dc11-8605-41c0-b705-26ef3c38f885\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") "
Oct 31 08:24:25.610294 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610180    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\") pod \"17b7dc11-8605-41c0-b705-26ef3c38f885\" (UID: \"17b7dc11-8605-41c0-b705-26ef3c38f885\") "
Oct 31 08:24:25.610446 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610392    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location" pod "17b7dc11-8605-41c0-b705-26ef3c38f885" (UID: "17b7dc11-8605-41c0-b705-26ef3c38f885"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:25.610488 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610442    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config" pod "17b7dc11-8605-41c0-b705-26ef3c38f885" (UID: "17b7dc11-8605-41c0-b705-26ef3c38f885"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:24:25.610540 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.610504    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert" pod "17b7dc11-8605-41c0-b705-26ef3c38f885" (UID: "17b7dc11-8605-41c0-b705-26ef3c38f885"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:24:25.612475 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.612447    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls" pod "17b7dc11-8605-41c0-b705-26ef3c38f885" (UID: "17b7dc11-8605-41c0-b705-26ef3c38f885"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:24:25.612622 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.612503    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq" pod "17b7dc11-8605-41c0-b705-26ef3c38f885" (UID: "17b7dc11-8605-41c0-b705-26ef3c38f885"). InnerVolumeSpecName "kube-api-access-cffrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:24:25.613453 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.613431    4371 generic.go:356] "Generic (PLEG): container finished" podID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerID="676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856" exitCode=0
Oct 31 08:24:25.613544 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.613475    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerDied","Data":"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"}
Oct 31 08:24:25.613592 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.613555    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"
Oct 31 08:24:25.613592 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.613587    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4" event={"ID":"17b7dc11-8605-41c0-b705-26ef3c38f885","Type":"ContainerDied","Data":"f0409b7a6f0886c9f7c7a2795fbdf90715e158d991c546c171dcab8d509fab27"}
Oct 31 08:24:25.613660 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.613621    4371 scope.go:122] "RemoveContainer" containerID="74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"
Oct 31 08:24:25.629811 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.629794    4371 scope.go:122] "RemoveContainer" containerID="676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"
Oct 31 08:24:25.637045 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.637024    4371 scope.go:122] "RemoveContainer" containerID="dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"
Oct 31 08:24:25.646734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.646530    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:24:25.648367 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.648344    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-mcp-predictor-698cffff67-gktj4"]
Oct 31 08:24:25.650671 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.650643    4371 scope.go:122] "RemoveContainer" containerID="f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"
Oct 31 08:24:25.658196 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.658173    4371 scope.go:122] "RemoveContainer" containerID="74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"
Oct 31 08:24:25.658456 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:25.658437    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14\": container with ID starting with 74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14 not found: ID does not exist" containerID="74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"
Oct 31 08:24:25.658519 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.658464    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14"} err="failed to get container status \"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14\": rpc error: code = NotFound desc = could not find container \"74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14\": container with ID starting with 74115df4ec85122ceb87e5e18d3acdbb861ee3b0cc94fbdb002a3a425b607e14 not found: ID does not exist"
Oct 31 08:24:25.658519 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.658489    4371 scope.go:122] "RemoveContainer" containerID="676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"
Oct 31 08:24:25.658770 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:25.658755    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856\": container with ID starting with 676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856 not found: ID does not exist" containerID="676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"
Oct 31 08:24:25.658828 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.658774    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856"} err="failed to get container status \"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856\": rpc error: code = NotFound desc = could not find container \"676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856\": container with ID starting with 676cb9c5fb7ceb1d2529eae587d2a13a4e8610a4fba0665ae33dd4edaa60a856 not found: ID does not exist"
Oct 31 08:24:25.658828 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.658788    4371 scope.go:122] "RemoveContainer" containerID="dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"
Oct 31 08:24:25.659021 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:25.659005    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2\": container with ID starting with dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2 not found: ID does not exist" containerID="dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"
Oct 31 08:24:25.659073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.659025    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2"} err="failed to get container status \"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2\": rpc error: code = NotFound desc = could not find container \"dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2\": container with ID starting with dcae7f251490295d1341be95d6b0671e04e9cab99e3b1d5d957b5408e65731f2 not found: ID does not exist"
Oct 31 08:24:25.659073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.659037    4371 scope.go:122] "RemoveContainer" containerID="f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"
Oct 31 08:24:25.659276 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:25.659262    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617\": container with ID starting with f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617 not found: ID does not exist" containerID="f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"
Oct 31 08:24:25.659317 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.659279    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617"} err="failed to get container status \"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617\": rpc error: code = NotFound desc = could not find container \"f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617\": container with ID starting with f3f9b6387ddadaa09485e3c39a0f9bbfb22da670dfb91b00484f09918432a617 not found: ID does not exist"
Oct 31 08:24:25.703328 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.703300    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" path="/var/lib/kubelet/pods/17b7dc11-8605-41c0-b705-26ef3c38f885/volumes"
Oct 31 08:24:25.711264 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.711242    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b7dc11-8605-41c0-b705-26ef3c38f885-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:25.711378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.711268    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:25.711378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.711285    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffrq\" (UniqueName: \"kubernetes.io/projected/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-api-access-cffrq\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:25.711378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.711298    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b7dc11-8605-41c0-b705-26ef3c38f885-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:25.711378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:25.711313    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b7dc11-8605-41c0-b705-26ef3c38f885-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:26.244072 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244038    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:24:26.244333 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244322    4371 cpu_manager.go:414] "containerMap: removing container" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-agent"
Oct 31 08:24:26.244378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244335    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-agent"
Oct 31 08:24:26.244378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244356    4371 cpu_manager.go:414] "containerMap: removing container" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="storage-initializer"
Oct 31 08:24:26.244378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244362    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="storage-initializer"
Oct 31 08:24:26.244378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244368    4371 cpu_manager.go:414] "containerMap: removing container" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container"
Oct 31 08:24:26.244378 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244373    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container"
Oct 31 08:24:26.244593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244390    4371 cpu_manager.go:414] "containerMap: removing container" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy"
Oct 31 08:24:26.244593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244395    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy"
Oct 31 08:24:26.244593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244439    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-container"
Oct 31 08:24:26.244593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244448    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kube-rbac-proxy"
Oct 31 08:24:26.244593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.244455    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b7dc11-8605-41c0-b705-26ef3c38f885" containerName="kserve-agent"
Oct 31 08:24:26.249029 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.249011    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.249923 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.249899    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:26.255903 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.255850    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:24:26.255976 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.255917    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 31 08:24:26.256206 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.256185    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:24:26.257576 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.257537    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:24:26.257699 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.257604    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:24:26.257699 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.257685    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:24:26.257822 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.257784    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-paddle-predictor-serving-cert\""
Oct 31 08:24:26.259760 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.259744    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:24:26.349645 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.349613    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:26.417458 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.417426    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.417458 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.417462    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdhr\" (UniqueName: \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.417690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.417559    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.417690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.417586    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.417690 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.417607    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.518542 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518446    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.518542 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518478    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.518542 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518496    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.518845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518561    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.518845 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518580    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdhr\" (UniqueName: \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.519010 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.518988    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.519206 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.519187    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.519264 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.519248    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.520817 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.520799    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.527018 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.526991    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdhr\" (UniqueName: \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\") pod \"isvc-paddle-predictor-6c54bc6b96-t5z4l\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") " pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.566955 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.566933    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:26.837349 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:26.837317    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:24:26.839659 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:24:26.839623    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac84764_6c3b_4fa4_b343_cb99bb3f143e.slice/crio-a7ee2d40e59cef8a0dd308bf6f12bee21cdcb558c9727ee802c10612f32b021e WatchSource:0}: Error finding container a7ee2d40e59cef8a0dd308bf6f12bee21cdcb558c9727ee802c10612f32b021e: Status 404 returned error can't find the container with id a7ee2d40e59cef8a0dd308bf6f12bee21cdcb558c9727ee802c10612f32b021e
Oct 31 08:24:27.624933 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:27.624890    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerStarted","Data":"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"}
Oct 31 08:24:27.625302 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:27.624977    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerStarted","Data":"a7ee2d40e59cef8a0dd308bf6f12bee21cdcb558c9727ee802c10612f32b021e"}
Oct 31 08:24:29.788496 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:29.788453    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:24:32.646954 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:32.646919    4371 generic.go:356] "Generic (PLEG): container finished" podID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerID="5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd" exitCode=0
Oct 31 08:24:32.647336 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:32.646975    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerDied","Data":"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"}
Oct 31 08:24:39.644931 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:39.644887    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:44.702293 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:44.702261    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerStarted","Data":"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"}
Oct 31 08:24:44.702737 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:44.702321    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerStarted","Data":"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"}
Oct 31 08:24:44.702737 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:44.702535    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:44.722809 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:44.722753    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podStartSLOduration=2.350479441 podStartE2EDuration="18.722740912s" totalImagesPullingTime="11.244350237s" totalInitContainerRuntime="5.127911234s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:24:44.720754088 +0000 UTC m=+3327.136884761" watchObservedRunningTime="2026-10-31 08:24:44.722740912 +0000 UTC m=+3327.138871584"
Oct 31 08:24:45.010345 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:45.010313    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:45.110341 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:45.110309    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:45.705376 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:45.705344    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:45.706570 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:45.706542    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:24:46.709305 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:46.709267    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:24:51.713608 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:51.713572    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:24:51.714075 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:24:51.714032    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:24:59.789495 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:24:59.789457    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:25:01.714562 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:01.714500    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:25:03.579419 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:03.579366    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:11.714583 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:11.714481    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:25:21.714049 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:21.714001    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Oct 31 08:25:29.789371 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:25:29.789336    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:25:31.714703 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:31.714625    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:25:37.987654 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:37.987622    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:25:37.988073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:37.987861    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container" containerID="cri-o://1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd" gracePeriod=30
Oct 31 08:25:37.988073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:37.987906    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kube-rbac-proxy" containerID="cri-o://326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543" gracePeriod=30
Oct 31 08:25:38.050769 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:38.050724    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:38.897966 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:38.897933    4371 generic.go:356] "Generic (PLEG): container finished" podID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerID="326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543" exitCode=0
Oct 31 08:25:38.898150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:38.898017    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerDied","Data":"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"}
Oct 31 08:25:40.823878 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.823857    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:25:40.907816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.907726    4371 generic.go:356] "Generic (PLEG): container finished" podID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerID="1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd" exitCode=0
Oct 31 08:25:40.907816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.907792    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"
Oct 31 08:25:40.907816 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.907800    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerDied","Data":"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"}
Oct 31 08:25:40.908086 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.907862    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l" event={"ID":"9ac84764-6c3b-4fa4-b343-cb99bb3f143e","Type":"ContainerDied","Data":"a7ee2d40e59cef8a0dd308bf6f12bee21cdcb558c9727ee802c10612f32b021e"}
Oct 31 08:25:40.908086 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.907898    4371 scope.go:122] "RemoveContainer" containerID="326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"
Oct 31 08:25:40.916053 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.916033    4371 scope.go:122] "RemoveContainer" containerID="1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"
Oct 31 08:25:40.923192 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.923176    4371 scope.go:122] "RemoveContainer" containerID="5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"
Oct 31 08:25:40.930122 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930098    4371 scope.go:122] "RemoveContainer" containerID="326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"
Oct 31 08:25:40.930386 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:25:40.930361    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543\": container with ID starting with 326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543 not found: ID does not exist" containerID="326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"
Oct 31 08:25:40.930450 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930396    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543"} err="failed to get container status \"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543\": rpc error: code = NotFound desc = could not find container \"326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543\": container with ID starting with 326262da4add226d7dc89593c70f519bcbe99d1e9f7749bdcae91e153078f543 not found: ID does not exist"
Oct 31 08:25:40.930450 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930411    4371 scope.go:122] "RemoveContainer" containerID="1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"
Oct 31 08:25:40.930682 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:25:40.930662    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd\": container with ID starting with 1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd not found: ID does not exist" containerID="1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"
Oct 31 08:25:40.930737 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930688    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd"} err="failed to get container status \"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd\": rpc error: code = NotFound desc = could not find container \"1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd\": container with ID starting with 1c5392780873b2888db241007adbbc331ae3b3bc64921797e6487f7e8a0405bd not found: ID does not exist"
Oct 31 08:25:40.930737 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930705    4371 scope.go:122] "RemoveContainer" containerID="5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"
Oct 31 08:25:40.930960 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:25:40.930937    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd\": container with ID starting with 5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd not found: ID does not exist" containerID="5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"
Oct 31 08:25:40.931012 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.930968    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd"} err="failed to get container status \"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd\": rpc error: code = NotFound desc = could not find container \"5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd\": container with ID starting with 5eae0876da56ca7391d8762bfe80a48355123012b82ba9d00754287a8b5306cd not found: ID does not exist"
Oct 31 08:25:40.982364 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982338    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\" (UniqueName: \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\") pod \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") "
Oct 31 08:25:40.982469 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982390    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\") pod \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") "
Oct 31 08:25:40.982469 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982433    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\") pod \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") "
Oct 31 08:25:40.982469 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982455    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\") pod \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") "
Oct 31 08:25:40.982619 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982475    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\") pod \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\" (UID: \"9ac84764-6c3b-4fa4-b343-cb99bb3f143e\") "
Oct 31 08:25:40.982822 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982799    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert" pod "9ac84764-6c3b-4fa4-b343-cb99bb3f143e" (UID: "9ac84764-6c3b-4fa4-b343-cb99bb3f143e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:25:40.982822 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.982810    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config" pod "9ac84764-6c3b-4fa4-b343-cb99bb3f143e" (UID: "9ac84764-6c3b-4fa4-b343-cb99bb3f143e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:25:40.984407 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.984382    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls" pod "9ac84764-6c3b-4fa4-b343-cb99bb3f143e" (UID: "9ac84764-6c3b-4fa4-b343-cb99bb3f143e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:25:40.984540 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.984521    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr" pod "9ac84764-6c3b-4fa4-b343-cb99bb3f143e" (UID: "9ac84764-6c3b-4fa4-b343-cb99bb3f143e"). InnerVolumeSpecName "kube-api-access-8bdhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:25:40.992037 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:40.992018    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location" pod "9ac84764-6c3b-4fa4-b343-cb99bb3f143e" (UID: "9ac84764-6c3b-4fa4-b343-cb99bb3f143e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:25:41.083753 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.083719    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdhr\" (UniqueName: \"kubernetes.io/projected/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-api-access-8bdhr\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:25:41.083753 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.083751    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:25:41.083923 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.083763    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:25:41.083923 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.083772    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:25:41.083923 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.083782    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac84764-6c3b-4fa4-b343-cb99bb3f143e-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:25:41.230526 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.230484    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:25:41.234042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.234019    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-paddle-predictor-6c54bc6b96-t5z4l"]
Oct 31 08:25:41.703680 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:41.703647    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" path="/var/lib/kubelet/pods/9ac84764-6c3b-4fa4-b343-cb99bb3f143e/volumes"
Oct 31 08:25:43.402699 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.402667    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:43.405627 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405596    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:25:43.405913 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405898    4371 cpu_manager.go:414] "containerMap: removing container" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container"
Oct 31 08:25:43.405967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405913    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container"
Oct 31 08:25:43.405967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405926    4371 cpu_manager.go:414] "containerMap: removing container" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kube-rbac-proxy"
Oct 31 08:25:43.405967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405931    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kube-rbac-proxy"
Oct 31 08:25:43.405967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405939    4371 cpu_manager.go:414] "containerMap: removing container" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="storage-initializer"
Oct 31 08:25:43.405967 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405944    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="storage-initializer"
Oct 31 08:25:43.406124 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405989    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kserve-container"
Oct 31 08:25:43.406124 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.405998    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac84764-6c3b-4fa4-b343-cb99bb3f143e" containerName="kube-rbac-proxy"
Oct 31 08:25:43.408774 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.408758    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.412949 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.412932    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Oct 31 08:25:43.416119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.416076    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:25:43.416119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.416082    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:25:43.416381 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.416360    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:25:43.417786 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.417759    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:25:43.417786 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.417781    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-pmml-predictor-serving-cert\""
Oct 31 08:25:43.417956 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.417866    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:25:43.419056 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.419041    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:25:43.505225 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.505190    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.505413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.505237    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.505413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.505259    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.505413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.505353    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.505413 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.505390    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtcn\" (UniqueName: \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.508204 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.508178    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:43.605908 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.605876    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtcn\" (UniqueName: \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.605931    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.605960    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.605978    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606060 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.605999    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606410 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.606391    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606806 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.606783    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.606898 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.606783    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.608464 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.608445    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.613985 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.613934    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtcn\" (UniqueName: \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\") pod \"isvc-pmml-predictor-768f6657bf-2ft6v\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") " pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.724539 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.724489    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:43.999451 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:43.999426    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:25:44.001536 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:25:44.001495    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb79a54_fc91_4644_a695_c3d45cf312ef.slice/crio-551fd550ca137c8c1d58e3ab5fce0bb2afd62c1c2642da644301509fdd22cdb5 WatchSource:0}: Error finding container 551fd550ca137c8c1d58e3ab5fce0bb2afd62c1c2642da644301509fdd22cdb5: Status 404 returned error can't find the container with id 551fd550ca137c8c1d58e3ab5fce0bb2afd62c1c2642da644301509fdd22cdb5
Oct 31 08:25:44.923041 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:44.923004    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerStarted","Data":"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"}
Oct 31 08:25:44.923458 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:44.923058    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerStarted","Data":"551fd550ca137c8c1d58e3ab5fce0bb2afd62c1c2642da644301509fdd22cdb5"}
Oct 31 08:25:48.939557 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:48.939520    4371 generic.go:356] "Generic (PLEG): container finished" podID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerID="92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812" exitCode=0
Oct 31 08:25:48.939557 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:48.939560    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerDied","Data":"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"}
Oct 31 08:25:49.945247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:49.945214    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerStarted","Data":"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"}
Oct 31 08:25:49.945653 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:49.945282    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerStarted","Data":"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"}
Oct 31 08:25:49.945653 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:49.945478    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:49.966495 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:49.966453    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podStartSLOduration=2.865877031 podStartE2EDuration="6.966441546s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.100564515s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:25:49.964685849 +0000 UTC m=+3392.380816522" watchObservedRunningTime="2026-10-31 08:25:49.966441546 +0000 UTC m=+3392.382572217"
Oct 31 08:25:50.949188 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:50.949153    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:50.950388 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:50.950359    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:25:51.953593 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:51.953554    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:25:56.958119 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:56.958091    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:25:56.958585 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:25:56.958561    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:25:59.789343 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:25:59.789308    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:26:03.626390 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:03.626337    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:06.959353 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:06.959315    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:09.416156 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:09.416118    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:16.958922 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:16.958881    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:26.958534 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:26.958465    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:29.789443 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:26:29.789407    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:26:36.959590 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:36.959545    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:36.978284 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:36.978255    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:42.248415 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:42.248384    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:46.959097 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:46.959056    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:49.848811 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:49.848756    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:49.948279 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:49.948247    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:56.959501 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:26:56.959460    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:26:59.789757 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:26:59.789719    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:27:03.654753 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:03.654664    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:06.958607 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:06.958568    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:27:16.959215 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:16.959178    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:27:25.259061 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:25.259029    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:27:25.259703 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:25.259363    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" containerID="cri-o://f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866" gracePeriod=30
Oct 31 08:27:25.259703 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:25.259409    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kube-rbac-proxy" containerID="cri-o://6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc" gracePeriod=30
Oct 31 08:27:25.322223 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:25.322188    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:26.309486 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:26.309454    4371 generic.go:356] "Generic (PLEG): container finished" podID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerID="6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc" exitCode=0
Oct 31 08:27:26.309889 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:26.309536    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerDied","Data":"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"}
Oct 31 08:27:26.954295 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:26.954252    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Oct 31 08:27:26.958565 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:26.958538    4371 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Oct 31 08:27:29.499236 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.499210    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:27:29.626282 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626202    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\") pod \"edb79a54-fc91-4644-a695-c3d45cf312ef\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") "
Oct 31 08:27:29.626282 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626262    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\" (UniqueName: \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\") pod \"edb79a54-fc91-4644-a695-c3d45cf312ef\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") "
Oct 31 08:27:29.626502 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626308    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\") pod \"edb79a54-fc91-4644-a695-c3d45cf312ef\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") "
Oct 31 08:27:29.626502 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626332    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\") pod \"edb79a54-fc91-4644-a695-c3d45cf312ef\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") "
Oct 31 08:27:29.626502 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626359    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\") pod \"edb79a54-fc91-4644-a695-c3d45cf312ef\" (UID: \"edb79a54-fc91-4644-a695-c3d45cf312ef\") "
Oct 31 08:27:29.626771 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626742    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert" pod "edb79a54-fc91-4644-a695-c3d45cf312ef" (UID: "edb79a54-fc91-4644-a695-c3d45cf312ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:27:29.626878 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626749    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location" pod "edb79a54-fc91-4644-a695-c3d45cf312ef" (UID: "edb79a54-fc91-4644-a695-c3d45cf312ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:27:29.626878 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.626788    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config" pod "edb79a54-fc91-4644-a695-c3d45cf312ef" (UID: "edb79a54-fc91-4644-a695-c3d45cf312ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:27:29.628329 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.628303    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn" pod "edb79a54-fc91-4644-a695-c3d45cf312ef" (UID: "edb79a54-fc91-4644-a695-c3d45cf312ef"). InnerVolumeSpecName "kube-api-access-wrtcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:27:29.628422 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.628401    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls" pod "edb79a54-fc91-4644-a695-c3d45cf312ef" (UID: "edb79a54-fc91-4644-a695-c3d45cf312ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:27:29.727344 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.727303    4371 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:27:29.727344 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.727340    4371 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb79a54-fc91-4644-a695-c3d45cf312ef-proxy-tls\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:27:29.727344 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.727351    4371 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb79a54-fc91-4644-a695-c3d45cf312ef-kserve-provision-location\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:27:29.727563 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.727360    4371 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb79a54-fc91-4644-a695-c3d45cf312ef-cabundle-cert\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:27:29.727563 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:29.727369    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtcn\" (UniqueName: \"kubernetes.io/projected/edb79a54-fc91-4644-a695-c3d45cf312ef-kube-api-access-wrtcn\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:27:29.789214 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:27:29.789176    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:27:30.325006 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.324967    4371 generic.go:356] "Generic (PLEG): container finished" podID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerID="f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866" exitCode=0
Oct 31 08:27:30.325212 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.325019    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerDied","Data":"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"}
Oct 31 08:27:30.325212 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.325053    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"
Oct 31 08:27:30.325212 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.325094    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v" event={"ID":"edb79a54-fc91-4644-a695-c3d45cf312ef","Type":"ContainerDied","Data":"551fd550ca137c8c1d58e3ab5fce0bb2afd62c1c2642da644301509fdd22cdb5"}
Oct 31 08:27:30.325212 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.325131    4371 scope.go:122] "RemoveContainer" containerID="6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"
Oct 31 08:27:30.333319 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.333304    4371 scope.go:122] "RemoveContainer" containerID="f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"
Oct 31 08:27:30.340486 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.340471    4371 scope.go:122] "RemoveContainer" containerID="92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"
Oct 31 08:27:30.344525 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.344465    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:27:30.347738 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.347723    4371 scope.go:122] "RemoveContainer" containerID="6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"
Oct 31 08:27:30.347995 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:27:30.347976    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc\": container with ID starting with 6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc not found: ID does not exist" containerID="6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"
Oct 31 08:27:30.348072 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.348007    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc"} err="failed to get container status \"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc\": rpc error: code = NotFound desc = could not find container \"6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc\": container with ID starting with 6ef0250495f6ef0f17fcc54ce7437dc161c447940ea2ca26b9f1c6ee4cdfcafc not found: ID does not exist"
Oct 31 08:27:30.348072 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.348030    4371 scope.go:122] "RemoveContainer" containerID="f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"
Oct 31 08:27:30.348361 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:27:30.348342    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866\": container with ID starting with f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866 not found: ID does not exist" containerID="f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"
Oct 31 08:27:30.348446 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.348368    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866"} err="failed to get container status \"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866\": rpc error: code = NotFound desc = could not find container \"f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866\": container with ID starting with f2cf703a6be934ce31b552633afa7b70227e7ce384214948656d278103cc6866 not found: ID does not exist"
Oct 31 08:27:30.348446 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.348388    4371 scope.go:122] "RemoveContainer" containerID="92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"
Oct 31 08:27:30.348759 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:27:30.348711    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812\": container with ID starting with 92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812 not found: ID does not exist" containerID="92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"
Oct 31 08:27:30.348759 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.348746    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812"} err="failed to get container status \"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812\": rpc error: code = NotFound desc = could not find container \"92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812\": container with ID starting with 92342624200e7235a8d6d7253b301b374c6c7881b9c0887c5eda075b3465a812 not found: ID does not exist"
Oct 31 08:27:30.350028 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.350011    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-pmml-predictor-768f6657bf-2ft6v"]
Oct 31 08:27:30.522921 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:30.522892    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:31.657922 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.657887    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bp57f/must-gather-ksvhk"]
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658173    4371 cpu_manager.go:414] "containerMap: removing container" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="storage-initializer"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658184    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="storage-initializer"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658194    4371 cpu_manager.go:414] "containerMap: removing container" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kube-rbac-proxy"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658199    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kube-rbac-proxy"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658227    4371 cpu_manager.go:414] "containerMap: removing container" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658232    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658295    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kserve-container"
Oct 31 08:27:31.658330 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.658305    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" containerName="kube-rbac-proxy"
Oct 31 08:27:31.662896 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.662879    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.667717 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.667694    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp57f\"/\"openshift-service-ca.crt\""
Oct 31 08:27:31.667717 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.667715    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp57f/must-gather-ksvhk"]
Oct 31 08:27:31.667907 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.667900    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp57f\"/\"kube-root-ca.crt\""
Oct 31 08:27:31.667951 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.667900    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bp57f\"/\"default-dockercfg-c7wkg\""
Oct 31 08:27:31.678913 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.678892    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:31.704360 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.704334    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb79a54-fc91-4644-a695-c3d45cf312ef" path="/var/lib/kubelet/pods/edb79a54-fc91-4644-a695-c3d45cf312ef/volumes"
Oct 31 08:27:31.741385 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.741361    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.741489 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.741469    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.773184 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.773153    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:31.842630 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.842598    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.842804 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.842693    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.843015 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.842994    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.850551 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.850529    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\") pod \"must-gather-ksvhk\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") " pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:31.985977 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:31.985946    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:27:32.255058 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:32.255035    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp57f/must-gather-ksvhk"]
Oct 31 08:27:32.257399 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:27:32.257376    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4fc1c5a_e0ba_458b_8e65_4cc84a620963.slice/crio-ae15a8e788372d39fd71dc12ce8d3083023e5fc287b225320f7b2db13668f190 WatchSource:0}: Error finding container ae15a8e788372d39fd71dc12ce8d3083023e5fc287b225320f7b2db13668f190: Status 404 returned error can't find the container with id ae15a8e788372d39fd71dc12ce8d3083023e5fc287b225320f7b2db13668f190
Oct 31 08:27:32.333109 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:32.333075    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp57f/must-gather-ksvhk" event={"ID":"d4fc1c5a-e0ba-458b-8e65-4cc84a620963","Type":"ContainerStarted","Data":"ae15a8e788372d39fd71dc12ce8d3083023e5fc287b225320f7b2db13668f190"}
Oct 31 08:27:38.558310 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:38.558259    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:38.661116 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:38.661083    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:39.363455 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:39.363414    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp57f/must-gather-ksvhk" event={"ID":"d4fc1c5a-e0ba-458b-8e65-4cc84a620963","Type":"ContainerStarted","Data":"0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047"}
Oct 31 08:27:39.363667 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:39.363465    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp57f/must-gather-ksvhk" event={"ID":"d4fc1c5a-e0ba-458b-8e65-4cc84a620963","Type":"ContainerStarted","Data":"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"}
Oct 31 08:27:39.379488 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:39.379438    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bp57f/must-gather-ksvhk" podStartSLOduration=2.110600687 podStartE2EDuration="8.379423354s" totalImagesPullingTime="6.268822667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:27:39.379019954 +0000 UTC m=+3501.795150625" watchObservedRunningTime="2026-10-31 08:27:39.379423354 +0000 UTC m=+3501.795554025"
Oct 31 08:27:58.441229 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:58.441197    4371 generic.go:356] "Generic (PLEG): container finished" podID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerID="6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3" exitCode=0
Oct 31 08:27:58.441786 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:58.441272    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp57f/must-gather-ksvhk" event={"ID":"d4fc1c5a-e0ba-458b-8e65-4cc84a620963","Type":"ContainerDied","Data":"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"}
Oct 31 08:27:58.441786 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:58.441678    4371 scope.go:122] "RemoveContainer" containerID="6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"
Oct 31 08:27:58.518573 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:58.518533    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bp57f_must-gather-ksvhk_d4fc1c5a-e0ba-458b-8e65-4cc84a620963/gather/0.log"
Oct 31 08:27:59.143759 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:59.143723    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:59.399147 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:59.399066    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:59.502270 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:27:59.502239    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:59.789040 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:27:59.789009    4371 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd96w" containerName="extractor"
Oct 31 08:28:02.361208 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:02.361182    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xgqwg_bb9197ad-22d3-434a-9575-e60c63fe330a/global-pull-secret-syncer/0.log"
Oct 31 08:28:02.498042 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:02.498008    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log"
Oct 31 08:28:02.646438 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:02.646365    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-69.us-east-2.compute.internal_7057eabb38a9074e4b218a9c4b8b8759/haproxy/0.log"
Oct 31 08:28:03.697980 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:03.697948    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:04.053694 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.053650    4371 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bp57f/must-gather-ksvhk"]
Oct 31 08:28:04.053900 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.053823    4371 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-bp57f/must-gather-ksvhk" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="copy" containerID="cri-o://0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047" gracePeriod=2
Oct 31 08:28:04.058044 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.058021    4371 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bp57f/must-gather-ksvhk"]
Oct 31 08:28:04.285729 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.285708    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bp57f_must-gather-ksvhk_d4fc1c5a-e0ba-458b-8e65-4cc84a620963/copy/0.log"
Oct 31 08:28:04.287247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.287225    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:28:04.289696 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:28:04.289672    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-ksvhk\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bp57f\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:28:04.422088 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.421998    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\") pod \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") "
Oct 31 08:28:04.422253 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.422092    4371 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\") pod \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\" (UID: \"d4fc1c5a-e0ba-458b-8e65-4cc84a620963\") "
Oct 31 08:28:04.423382 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.423361    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output" pod "d4fc1c5a-e0ba-458b-8e65-4cc84a620963" (UID: "d4fc1c5a-e0ba-458b-8e65-4cc84a620963"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:04.424246 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.424212    4371 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d" pod "d4fc1c5a-e0ba-458b-8e65-4cc84a620963" (UID: "d4fc1c5a-e0ba-458b-8e65-4cc84a620963"). InnerVolumeSpecName "kube-api-access-jlf7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:28:04.466158 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.466133    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bp57f_must-gather-ksvhk_d4fc1c5a-e0ba-458b-8e65-4cc84a620963/copy/0.log"
Oct 31 08:28:04.466539 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.466494    4371 generic.go:356] "Generic (PLEG): container finished" podID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerID="0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047" exitCode=143
Oct 31 08:28:04.466635 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.466561    4371 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:28:04.466635 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.466590    4371 scope.go:122] "RemoveContainer" containerID="0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047"
Oct 31 08:28:04.469197 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:28:04.469164    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-ksvhk\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bp57f\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:28:04.474811 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.474792    4371 scope.go:122] "RemoveContainer" containerID="6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"
Oct 31 08:28:04.477357 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:28:04.477337    4371 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-ksvhk\" is forbidden: User \"system:node:ip-10-0-137-69.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bp57f\": no relationship found between node 'ip-10-0-137-69.us-east-2.compute.internal' and this object" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" pod="openshift-must-gather-bp57f/must-gather-ksvhk"
Oct 31 08:28:04.486000 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.485982    4371 scope.go:122] "RemoveContainer" containerID="0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047"
Oct 31 08:28:04.486275 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:28:04.486257    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047\": container with ID starting with 0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047 not found: ID does not exist" containerID="0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047"
Oct 31 08:28:04.486316 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.486284    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047"} err="failed to get container status \"0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047\": rpc error: code = NotFound desc = could not find container \"0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047\": container with ID starting with 0b980a0b14cba1f11e5a3248e4b874ec57be9fabf90c309537ce785d3b6aa047 not found: ID does not exist"
Oct 31 08:28:04.486316 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.486300    4371 scope.go:122] "RemoveContainer" containerID="6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"
Oct 31 08:28:04.486583 ip-10-0-137-69 kubenswrapper[4371]: E1031 08:28:04.486564    4371 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3\": container with ID starting with 6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3 not found: ID does not exist" containerID="6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"
Oct 31 08:28:04.486618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.486590    4371 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3"} err="failed to get container status \"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3\": rpc error: code = NotFound desc = could not find container \"6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3\": container with ID starting with 6386e3d89b3d8799a284f20640c05d346572604a0d41437067369a60e4bf19b3 not found: ID does not exist"
Oct 31 08:28:04.523391 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.523366    4371 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlf7d\" (UniqueName: \"kubernetes.io/projected/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-kube-api-access-jlf7d\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:28:04.523391 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.523389    4371 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4fc1c5a-e0ba-458b-8e65-4cc84a620963-must-gather-output\") on node \"ip-10-0-137-69.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:28:04.641461 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.641432    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:04.754247 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:04.754221    4371 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dvbr_67890a7c-54d4-4af5-a01c-fee014b737a8/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:05.705134 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:05.705106    4371 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" path="/var/lib/kubelet/pods/d4fc1c5a-e0ba-458b-8e65-4cc84a620963/volumes"
Oct 31 08:28:08.002052 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.002025    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vqspw_13f79276-4681-47fc-8a67-91dc6ec86c90/monitoring-plugin/0.log"
Oct 31 08:28:08.121278 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.121253    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrgr9_d8a954e8-34d3-46a8-89a9-0ec7e005d620/node-exporter/0.log"
Oct 31 08:28:08.144734 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.144709    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrgr9_d8a954e8-34d3-46a8-89a9-0ec7e005d620/kube-rbac-proxy/0.log"
Oct 31 08:28:08.171653 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.171629    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrgr9_d8a954e8-34d3-46a8-89a9-0ec7e005d620/init-textfile/0.log"
Oct 31 08:28:08.291568 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.291492    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6wbfw_7c2e8f6a-5c3b-4404-b98f-c0e3519d8841/kube-rbac-proxy-main/0.log"
Oct 31 08:28:08.316601 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.316579    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6wbfw_7c2e8f6a-5c3b-4404-b98f-c0e3519d8841/kube-rbac-proxy-self/0.log"
Oct 31 08:28:08.342420 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.342397    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6wbfw_7c2e8f6a-5c3b-4404-b98f-c0e3519d8841/openshift-state-metrics/0.log"
Oct 31 08:28:08.393895 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.393873    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/prometheus/0.log"
Oct 31 08:28:08.417203 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.417181    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/config-reloader/0.log"
Oct 31 08:28:08.447029 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.447007    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/thanos-sidecar/0.log"
Oct 31 08:28:08.474109 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.474087    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/kube-rbac-proxy-web/0.log"
Oct 31 08:28:08.502872 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.502847    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/kube-rbac-proxy/0.log"
Oct 31 08:28:08.530697 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.530678    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/kube-rbac-proxy-thanos/0.log"
Oct 31 08:28:08.560994 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.560937    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1ecd0177-02e1-497c-9a18-abb68932318b/init-config-reloader/0.log"
Oct 31 08:28:08.598726 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.598704    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nnmtp_32078f71-611e-416e-aab0-02709b28be86/prometheus-operator/0.log"
Oct 31 08:28:08.633972 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.633953    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nnmtp_32078f71-611e-416e-aab0-02709b28be86/kube-rbac-proxy/0.log"
Oct 31 08:28:08.665067 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.665045    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8z45m_ca5afcaf-57e8-4957-a9ce-766f71811769/prometheus-operator-admission-webhook/0.log"
Oct 31 08:28:08.810281 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.810253    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/thanos-query/0.log"
Oct 31 08:28:08.834817 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.834758    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/kube-rbac-proxy-web/0.log"
Oct 31 08:28:08.860488 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.860464    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/kube-rbac-proxy/0.log"
Oct 31 08:28:08.884989 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.884967    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/prom-label-proxy/0.log"
Oct 31 08:28:08.909738 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.909717    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/kube-rbac-proxy-rules/0.log"
Oct 31 08:28:08.938195 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:08.938170    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d78f7778-sxmhc_64c4a477-81cf-495f-bcfe-ffc53cf532e8/kube-rbac-proxy-metrics/0.log"
Oct 31 08:28:10.329937 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.329906    4371 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"]
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330200    4371 cpu_manager.go:414] "containerMap: removing container" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="copy"
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330211    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="copy"
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330222    4371 cpu_manager.go:414] "containerMap: removing container" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="gather"
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330227    4371 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="gather"
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330278    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="copy"
Oct 31 08:28:10.330423 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.330285    4371 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fc1c5a-e0ba-458b-8e65-4cc84a620963" containerName="gather"
Oct 31 08:28:10.333398 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.333381    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.342561 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.342535    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"]
Oct 31 08:28:10.429150 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.429120    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jbks9\"/\"kube-root-ca.crt\""
Oct 31 08:28:10.429322 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.429167    4371 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jbks9\"/\"openshift-service-ca.crt\""
Oct 31 08:28:10.429322 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.429215    4371 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jbks9\"/\"default-dockercfg-dpgr6\""
Oct 31 08:28:10.465589 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.465565    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-lib-modules\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.465723 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.465624    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-sys\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.465765 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.465725    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-podres\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.465807 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.465767    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-proc\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.465807 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.465785    4371 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4d6\" (UniqueName: \"kubernetes.io/projected/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-kube-api-access-fn4d6\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566636 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566605    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-proc\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566636 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566635    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4d6\" (UniqueName: \"kubernetes.io/projected/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-kube-api-access-fn4d6\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566668    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-lib-modules\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566711    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-sys\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566732    4371 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-podres\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566734    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-proc\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566826    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-podres\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.566866 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566862    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-sys\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.567078 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.566880    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-lib-modules\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.574874 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.574846    4371 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4d6\" (UniqueName: \"kubernetes.io/projected/84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6-kube-api-access-fn4d6\") pod \"perf-node-gather-daemonset-rmvgc\" (UID: \"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.644945 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.644879    4371 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:10.918161 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:10.918067    4371 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"]
Oct 31 08:28:10.920427 ip-10-0-137-69 kubenswrapper[4371]: W1031 08:28:10.920401    4371 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84ee1c6d_688e_4b1c_8862_3b9dfaca1cd6.slice/crio-78bacbed2bfac2c6cdd94ee04182933ea45845a88ba1e96f0525941c527e53e9 WatchSource:0}: Error finding container 78bacbed2bfac2c6cdd94ee04182933ea45845a88ba1e96f0525941c527e53e9: Status 404 returned error can't find the container with id 78bacbed2bfac2c6cdd94ee04182933ea45845a88ba1e96f0525941c527e53e9
Oct 31 08:28:11.075269 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:11.075243    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7s9kp_317e3dea-65df-4a6a-917b-088d12b5fef3/download-server/0.log"
Oct 31 08:28:11.494303 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:11.494265    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc" event={"ID":"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6","Type":"ContainerStarted","Data":"7382c7ff72f0fafcc8544d8381e5543abed7a521b77675a1a37a76679b364ce8"}
Oct 31 08:28:11.494828 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:11.494324    4371 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc" event={"ID":"84ee1c6d-688e-4b1c-8862-3b9dfaca1cd6","Type":"ContainerStarted","Data":"78bacbed2bfac2c6cdd94ee04182933ea45845a88ba1e96f0525941c527e53e9"}
Oct 31 08:28:11.494828 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:11.494400    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:11.511537 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:11.511454    4371 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc" podStartSLOduration=1.511436095 podStartE2EDuration="1.511436095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:28:11.510473233 +0000 UTC m=+3533.926603903" watchObservedRunningTime="2026-10-31 08:28:11.511436095 +0000 UTC m=+3533.927566768"
Oct 31 08:28:12.662225 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:12.662199    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-llxtn_2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4/dns/0.log"
Oct 31 08:28:12.692073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:12.692042    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-llxtn_2c8200a7-46cd-4b23-a1f2-1acdb0dd67d4/kube-rbac-proxy/0.log"
Oct 31 08:28:12.849001 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:12.848974    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xg255_42302c13-04fb-415c-affb-f652ff0be51c/dns-node-resolver/0.log"
Oct 31 08:28:13.323870 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:13.323836    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65ddfff67-pdktj_bb292afd-74af-4692-ab45-c834ed320600/registry/0.log"
Oct 31 08:28:13.374693 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:13.374665    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkwsp_b89ab622-9b1b-4954-924b-b75da49f162f/node-ca/0.log"
Oct 31 08:28:14.628913 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:14.628881    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x2pmn_6eb4776c-2843-4b41-ab1e-1e65fa89ff36/serve-healthcheck-canary/0.log"
Oct 31 08:28:15.210466 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:15.210440    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd96w_0b2784ec-df9b-4a2e-8dc6-4af7d170341a/kube-rbac-proxy/0.log"
Oct 31 08:28:15.235644 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:15.235621    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd96w_0b2784ec-df9b-4a2e-8dc6-4af7d170341a/exporter/0.log"
Oct 31 08:28:15.260717 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:15.260691    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd96w_0b2784ec-df9b-4a2e-8dc6-4af7d170341a/extractor/0.log"
Oct 31 08:28:17.507384 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:17.507350    4371 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rmvgc"
Oct 31 08:28:18.195329 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.195291    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pt9kz_3c2df43b-c794-4d6e-aaee-310981189dbd/s3-init/0.log"
Oct 31 08:28:18.215083 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.215057    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pt9kz_3c2df43b-c794-4d6e-aaee-310981189dbd/download-hf-model/0.log"
Oct 31 08:28:18.246142 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.246118    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sl7pb_eeed32ef-11b2-4a38-8acc-20e06f4fcbd2/s3-tls-init-custom/0.log"
Oct 31 08:28:18.267073 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.267050    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sl7pb_eeed32ef-11b2-4a38-8acc-20e06f4fcbd2/download-hf-model/0.log"
Oct 31 08:28:18.301154 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.301135    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-5kbwc_925df15d-1949-4868-875d-0f49de11fd8d/s3-tls-init-serving/0.log"
Oct 31 08:28:18.321651 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:18.321628    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-5kbwc_925df15d-1949-4868-875d-0f49de11fd8d/download-hf-model/0.log"
Oct 31 08:28:24.234480 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.234454    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/kube-multus-additional-cni-plugins/0.log"
Oct 31 08:28:24.260186 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.260167    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/egress-router-binary-copy/0.log"
Oct 31 08:28:24.284295 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.284277    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/cni-plugins/0.log"
Oct 31 08:28:24.308041 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.308019    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/bond-cni-plugin/0.log"
Oct 31 08:28:24.332365 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.332337    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/routeoverride-cni/0.log"
Oct 31 08:28:24.383276 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.383256    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/whereabouts-cni-bincopy/0.log"
Oct 31 08:28:24.402490 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.402472    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gxcql_a1503777-02b6-4287-86f7-e1b7094397f2/whereabouts-cni/0.log"
Oct 31 08:28:24.624618 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.624549    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gt6sx_ccd6ee2e-4015-471d-9fe6-b9efc31594ae/kube-multus/0.log"
Oct 31 08:28:24.771087 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.771058    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6glq2_41668c10-19d0-4d52-b138-81c2dea94d7f/network-metrics-daemon/0.log"
Oct 31 08:28:24.793289 ip-10-0-137-69 kubenswrapper[4371]: I1031 08:28:24.793267    4371 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6glq2_41668c10-19d0-4d52-b138-81c2dea94d7f/kube-rbac-proxy/0.log"